diff --git a/README.md b/README.md index b38445b..cdfaa03 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Includes `synapsectl` command line utility: % synapsectl --help usage: synapsectl [-h] [--uri URI] [--version] [--verbose] - {discover,info,query,start,stop,configure,logs,read,plot,file,taps,deploy,build,settings} ... + {discover,info,query,start,stop,configure,logs,read,plot,file,taps,deploy,build,settings,deploy-model} ... Synapse Device Manager @@ -17,7 +17,7 @@ Includes `synapsectl` command line utility: --verbose, -v Enable verbose output Commands: - {discover,info,query,start,stop,configure,logs,read,plot,file,taps,deploy,build,settings} + {discover,info,query,start,stop,configure,logs,read,plot,file,taps,deploy,build,settings,deploy-model} discover Discover Synapse devices on the network info Get device information query Execute a query on the device @@ -32,6 +32,7 @@ Includes `synapsectl` command line utility: deploy Deploy an application to a Synapse device build Cross-compile and package an application into a .deb without deploying settings Manage the persistent device settings + deploy-model Deploy a machine learning model to a Synapse device As well as the base for a device implementation (`synapse/server`), @@ -194,3 +195,123 @@ After recording data to a file, you can generate plots to visualize your data. U ``` synapsectl plot --dir ``` + +## Model Deployment + +Deploy machine learning models to Synapse devices. + +### Prerequisites + +1. **Docker** — required for model conversion +2. **QAIRT SDK v2.34** (Qualcomm AI Runtime) — required for model conversion for DSP runtime, not required for CPU runtime + +### Quick Start — Deploy a Float Model (CPU) + +The simplest path — no calibration data needed, runs on CPU via onnxruntime: + +```bash +synapsectl deploy-model model.onnx \ + --name my_model \ + -u +``` + +#### Installing the QAIRT SDK + +1. Create a free account at [softwarecenter.qualcomm.com](https://softwarecenter.qualcomm.com/) (no paid license required) +2. Download **Qualcomm Software Center** (Linux `.deb`) and **Qualcomm AI Runtime v2.34** (Linux `.qik`) from the website +3. Install both: + ```bash + # Install Qualcomm Software Center (includes the qik package manager) + sudo dpkg -i QualcommSoftwareCenter*.deb + + # Install the QAIRT SDK + sudo /opt/qcom/softwarecenter/bin/qik/qik INSTALL "/path/to/Qualcomm_AI_Runtime_SDK.2.34.0.250424.Linux-AnyCPU.qik" + ``` +4. The SDK installs to `/opt/qcom/aistack/qairt/2.34.0.250424`. You'll pass this path as `--snpe-root` when deploying models. + + +### Deploy a Quantized Model (DSP) + +For production performance, quantize the model to INT8 for DSP inference. This requires **calibration data** — a small set of example inputs that represent what the model will see in real use. + +#### What is calibration data and why do I need it? + +Quantization converts your model from 32-bit floats to 8-bit integers, making it ~4x smaller and much faster on the DSP. But to do this well, the quantizer needs to see what typical input values look like so it can choose the right scale for each layer. Bad calibration data leads to accuracy loss. + +**Good calibration data** is a handful of real (or realistic) inputs from your application. For example: +- If your model processes neural signals, use 10-50 snippets of actual recorded neural data +- If your model processes audio, use 10-50 clips of real audio +- If you don't have real data yet, synthetic data that matches the expected distribution is acceptable + +Ideally, use at least **1000 representative samples** for best accuracy. Fewer samples (50-100) can work for initial testing, but more data gives the quantizer a better picture of your model's value ranges. + +#### Step 1: Create calibration `.raw` files + +Each `.raw` file is a flat binary dump of float32 values matching your model's input shape. Create them with numpy: + +```python +import numpy as np + +# Example: model expects input shape [1, 1920] +# Load your real data here — these should be actual inputs, not random noise +for i, sample_data in enumerate(my_real_samples[:20]): + # sample_data should be a numpy array with shape matching your model input + sample_data.astype(np.float32).tofile(f"sample_{i:03d}.raw") +``` + +If you don't have real data yet, you can use synthetic data to get started (accuracy may be lower): + +```python +import numpy as np + +for i in range(20): + sample = np.random.randn(1, 1920).astype(np.float32) + sample.tofile(f"sample_{i:03d}.raw") +``` + +#### Step 2: Create an input list file + +Create a text file called `input_list.txt` listing your `.raw` files, one per line. Put this file in the same directory as the `.raw` files. + +``` +sample_000.raw +sample_001.raw +sample_002.raw +sample_003.raw +sample_004.raw +sample_005.raw +sample_006.raw +sample_007.raw +sample_008.raw +sample_009.raw +``` + +#### Step 3: Deploy with quantization + +```bash +synapsectl deploy-model model.onnx \ + --name my_model \ + --quantize --input-list input_list.txt \ + --snpe-root /opt/qcom/aistack/qairt/2.34.0.250424 \ + -u +``` + +### Use in Your C++ App + +```cpp +#include + +// Loads models/.dlc from the device model directory +auto model = synapse::create_model("my_model"); + +if (model && model->is_ready()) { + auto result = model->infer(input_data); + // result.success, result.outputs, result.inference_time_us +} +``` + +The runtime is selected automatically: quantized models run on the DSP, float models run on CPU. You can also specify a runtime explicitly: + +```cpp +auto model = synapse::create_model("my_model", synapse::InferenceRuntime::kDsp); +``` diff --git a/setup.py b/setup.py index 263df0f..d86cbcb 100644 --- a/setup.py +++ b/setup.py @@ -6,11 +6,17 @@ setup( name="science-synapse", - version="2.5.0", + version="2.6.0a2", description="Client library and CLI for the Synapse API", author="Science Team", author_email="team@science.xyz", packages=find_packages(include=["synapse", "synapse.*"]), + package_data={ + "synapse": [ + "utils/model_converter/docker/Dockerfile", + "utils/model_converter/docker/convert.py", + ], + }, long_description=long_description, long_description_content_type="text/markdown", python_requires=">=3.9", @@ -32,6 +38,14 @@ "scipy", "h5py", ], + extras_require={ + "model-convert": [ + "onnx>=1.12.0,<1.16.0", + "torch", + "packaging", + "protobuf", + ], + }, entry_points={ "console_scripts": [ "synapsectl = synapse.cli:main", diff --git a/synapse/cli/__main__.py b/synapse/cli/__main__.py index af66f34..b9fc880 100755 --- a/synapse/cli/__main__.py +++ b/synapse/cli/__main__.py @@ -10,6 +10,7 @@ from synapse.cli import ( apps, + deploy_model, discover, files, offline_plot, @@ -77,6 +78,7 @@ def main(): taps.add_commands(subparsers) apps.add_commands(subparsers) settings.add_commands(subparsers) + deploy_model.add_commands(subparsers) args = parser.parse_args() # If we need to setup the device URI, do that now diff --git a/synapse/cli/build.py b/synapse/cli/build.py index a76fb8c..cdda8ae 100644 --- a/synapse/cli/build.py +++ b/synapse/cli/build.py @@ -340,6 +340,11 @@ def build_deb_package(app_dir: str, app_name: str, version: str = "0.1.0") -> bo lib_dst_dir = os.path.join(staging_dir, "opt", "scifi", "lib") os.makedirs(lib_dst_dir, exist_ok=True) + # QNN libraries are dlopen'd from /usr/lib/ (hardcoded paths in SDK), + # so they must be staged there — not in /opt/scifi/lib/ + qnn_dst_dir = os.path.join(staging_dir, "usr", "lib") + os.makedirs(qnn_dst_dir, exist_ok=True) + try: arch_suffix = detect_arch() image_tag = f"{app_name}:latest-{arch_suffix}" @@ -349,6 +354,33 @@ def build_deb_package(app_dir: str, app_name: str, version: str = "0.1.0") -> bo f"[yellow]Extracting SDK libraries from Docker image [bold]{image_tag}[/bold]...[/yellow]" ) + # Skip vcpkg .so files already shipped by scifi-headstage-shared-libraries + # to avoid dpkg file-overwrite conflicts when the app deb is installed. + extract_script = r""" +set -e +filter=/tmp/scifi_shared_libs.txt +: > "$filter" +if dpkg-query -W -f='${Status}' scifi-headstage-shared-libraries 2>/dev/null | grep -q "install ok installed"; then + dpkg-query -L scifi-headstage-shared-libraries | grep -oE '[^/]+\.so[^/]*$' | sort -u > "$filter" || true +else + echo "WARNING: scifi-headstage-shared-libraries not installed in build image; packaging all vcpkg .so files (may conflict on device)" >&2 +fi + +find /usr/lib -maxdepth 1 -name 'libsynapse*.so*' -exec cp -a {} /out/ \; + +vcpkg_lib="${VCPKG_ROOT}/build/host/vcpkg_installed/arm64-linux-dynamic-release/lib" +if [ -d "$vcpkg_lib" ]; then + find "$vcpkg_lib" -maxdepth 1 -name '*.so*' -print0 | while IFS= read -r -d '' f; do + base=$(basename "$f") + if [ -s "$filter" ] && grep -qxF "$base" "$filter"; then + echo "Skipping $base (already shipped by scifi-headstage-shared-libraries)" >&2 + else + cp -a "$f" /out/ + fi + done +fi +""".strip() + docker_cmd = [ "docker", "run", @@ -360,11 +392,29 @@ def build_deb_package(app_dir: str, app_name: str, version: str = "0.1.0") -> bo image_tag, "/bin/bash", "-c", - "find /usr/lib -name 'libsynapse*.so*' -exec cp -a {} /out/ \\;", + extract_script, ] subprocess.run(docker_cmd, check=True) + # Extract QNN runtime libraries → /usr/lib/ (dlopen'd by absolute path) + docker_cmd_qnn = [ + "docker", + "run", + "--rm", + "--platform", + platform_opt, + "-v", + f"{qnn_dst_dir}:/out", + image_tag, + "/bin/bash", + "-c", + "find /usr/lib -maxdepth 1 -name 'libQnn*.so' -exec cp -a {} /out/ \\; && " + "if [ -d /usr/lib/rfsa ]; then cp -a /usr/lib/rfsa /out/; fi", + ] + + subprocess.run(docker_cmd_qnn, check=True) + except subprocess.CalledProcessError as exc: console.print( f"[bold red]Error:[/bold red] Failed to copy SDK libraries from Docker image: {exc}" diff --git a/synapse/cli/deploy_model.py b/synapse/cli/deploy_model.py new file mode 100644 index 0000000..f63bfa3 --- /dev/null +++ b/synapse/cli/deploy_model.py @@ -0,0 +1,356 @@ +"""CLI command for deploying models to Synapse devices.""" + +import argparse +import os +from typing import Optional + +from rich.console import Console +from rich import progress +from rich.prompt import Confirm + +import synapse.client.sftp as sftp +from synapse.cli.files import setup_connection + +# Constants +DEVICE_MODEL_DIR = "/models" +DEFAULT_SFTP_USER = "scifi-sftp" +DEFAULT_ENV_FILE = ".scienv" + + +def add_commands(subparsers: argparse._SubParsersAction): + """Add the deploy-model command to the CLI.""" + parser = subparsers.add_parser( + "deploy-model", + help="Deploy a machine learning model to a Synapse device", + ) + + parser.add_argument( + "model_path", + type=str, + help="Path to the model file (.pt, .onnx, or .dlc)", + ) + + parser.add_argument( + "--input-shape", + type=str, + default=None, + help='Input shape for the model (e.g., "1,32,64"). Required if model has dynamic dimensions.', + ) + + parser.add_argument( + "--name", + type=str, + default=None, + help="Model name on device (default: filename without extension, e.g., 'my_model')", + ) + + parser.add_argument( + "--username", + type=str, + default=DEFAULT_SFTP_USER, + help=f"SFTP username (default: {DEFAULT_SFTP_USER})", + ) + + parser.add_argument( + "--env-file", + "-e", + type=str, + default=DEFAULT_ENV_FILE, + help=f"Password env file (default: {DEFAULT_ENV_FILE})", + ) + + parser.add_argument( + "--forget-password", + "-f", + action="store_true", + help="Don't store password locally", + ) + + parser.add_argument( + "--snpe-root", + type=str, + default=None, + help="Path to SNPE/QAIRT SDK root (or set SNPE_ROOT env var)", + ) + + parser.add_argument( + "--quantize", + action="store_true", + help=( + "Quantize the model to INT8 for DSP inference. Requires --input-list with " + "representative input samples. Quantized models run on the HTP/DSP backend " + "for maximum performance (~1ms). Without quantization, models run on CPU." + ), + ) + + parser.add_argument( + "--input-list", + type=str, + default=None, + help=( + "Path to a text file listing representative input samples for INT8 quantization. " + "Each line is a path to a .raw file (float32 binary). Required with --quantize. " + "Generate .raw files with: arr.astype(np.float32).tofile('sample.raw')" + ), + ) + + parser.add_argument( + "--force", + action="store_true", + help="Overwrite existing model on device without prompting", + ) + + parser.set_defaults(func=deploy_model) + + +def deploy_model(args): + """Deploy a model to a Synapse device.""" + console = Console() + + # Validate model path + if not os.path.exists(args.model_path): + console.print(f"[bold red]Error:[/bold red] Model file not found: {args.model_path}") + return + + # Parse input shape if provided + input_shape = None + if args.input_shape: + try: + input_shape = tuple(int(x.strip()) for x in args.input_shape.split(",")) + except ValueError: + console.print( + f"[bold red]Error:[/bold red] Invalid input shape format: {args.input_shape}" + ) + console.print('[yellow]Expected format: "dim1,dim2,..." (e.g., "1,32,64")[/yellow]') + return + + # Default dynamic dimensions to 1 if the model has them and no --input-shape given + if input_shape is None: + ext = os.path.splitext(args.model_path)[1].lower() + if ext == ".onnx": + try: + import onnx + + onnx_model = onnx.load(args.model_path) + for inp in onnx_model.graph.input: + dims = inp.type.tensor_type.shape.dim + has_dynamic = any(d.dim_param or d.dim_value == 0 for d in dims) + if has_dynamic: + resolved = [] + for d in dims: + if d.dim_param or d.dim_value == 0: + resolved.append(1) + else: + resolved.append(d.dim_value) + input_shape = tuple(resolved) + console.print( + f"[yellow]Note: model has dynamic dimensions, " + f"defaulting to {input_shape}[/yellow]" + ) + break + except Exception: + pass # If onnx isn't installed or can't load, let the converter handle it + + # Default model name to filename without extension + model_name = args.name + if model_name is None: + model_name = os.path.splitext(os.path.basename(args.model_path))[0] + quantize = args.quantize + + # Validate quantize + input-list + if quantize and not args.input_list: + console.print( + "[bold red]Error:[/bold red] --quantize requires --input-list " + "with representative input samples for INT8 calibration." + ) + console.print() + console.print("[dim]Example:[/dim]") + console.print(" synapsectl deploy-model model.onnx --name my_model \\") + console.print(" --quantize --input-list calibration_data.txt \\") + console.print(" --snpe-root /path/to/qairt/2.34.0.250424 -u ") + return + + if quantize: + fmt_str = "Quantized DLC (INT8) — runs on DSP" + else: + fmt_str = "ONNX (float32) — runs on CPU via ONNX Runtime" + + console.print(f"[bold]Deploying model:[/bold] {model_name}") + console.print(f"[bold]Source:[/bold] {args.model_path}") + console.print(f"[bold]Format:[/bold] {fmt_str}") + console.print() + + # Step 1: Prepare model for deployment + if quantize: + # Quantized path: convert to DLC via Docker (requires QAIRT SDK) + console.print("[bold cyan]Converting model to quantized DLC...[/bold cyan]") + + from synapse.utils.model_converter import convert_to_dlc + + dlc_path = convert_to_dlc( + args.model_path, + input_shape=input_shape, + snpe_root=args.snpe_root, + quantize=quantize, + input_list=args.input_list, + console=console, + ) + + if dlc_path is None: + console.print("[bold red]Model conversion failed[/bold red]") + return + + deploy_path = dlc_path + remote_ext = ".dlc" + else: + # Non-quantized path: deploy .onnx directly (no QAIRT SDK needed) + ext = os.path.splitext(args.model_path)[1].lower() + + if ext == ".dlc": + deploy_path = args.model_path + remote_ext = ".dlc" + elif ext == ".pt": + console.print("[bold cyan]Converting PyTorch model to ONNX...[/bold cyan]") + from synapse.utils.model_converter.pt_to_onnx import convert_pt_to_onnx + + onnx_path = convert_pt_to_onnx( + args.model_path, + input_shape=input_shape, + console=console, + ) + if onnx_path is None: + console.print("[bold red]Model conversion failed[/bold red]") + return + + deploy_path = onnx_path + remote_ext = ".onnx" + elif ext == ".onnx": + deploy_path = args.model_path + remote_ext = ".onnx" + else: + console.print(f"[bold red]Error:[/bold red] Unsupported file type: {ext}") + console.print("[yellow]Supported formats: .pt, .onnx, .dlc[/yellow]") + return + + console.print() + + # Step 2: Connect to device via SFTP + console.print("[bold cyan]Connecting to device...[/bold cyan]") + + result = setup_connection( + args.uri, + args.username, + args.env_file, + args.forget_password, + console, + ) + + if result is None: + return + + ssh, sftp_conn = result + + try: + # Step 3: Ensure model directory exists + _ensure_model_dir(sftp_conn, ssh, console) + + # Step 4: Check if model already exists on device + remote_path = f"{DEVICE_MODEL_DIR}/{model_name}{remote_ext}" + try: + sftp_conn.stat(remote_path) + if not args.force: + if not Confirm.ask( + f"[yellow]Model '{model_name}{remote_ext}' already exists on device. Overwrite?[/yellow]", + default=False, + ): + console.print("[dim]Aborted.[/dim]") + return + except FileNotFoundError: + pass + + # Step 5: Upload the model file + _upload_file(sftp_conn, deploy_path, remote_path, console) + + console.print() + console.print("[bold green]Model deployed successfully![/bold green]") + console.print() + console.print(f" Model deployed: [cyan]models/{model_name}{remote_ext}[/cyan]") + if quantize: + console.print(f" Runtime: [cyan]DSP (quantized INT8)[/cyan]") + else: + console.print(f" Runtime: [cyan]CPU (float32, ONNX Runtime)[/cyan]") + console.print() + console.print( + " [dim]Tip: for faster DSP inference (~1ms), redeploy with --quantize --input-list[/dim]" + ) + console.print() + console.print(" To load in your app:") + console.print(f' [cyan]auto model = synapse::create_model("{model_name}");[/cyan]') + + finally: + sftp.close_sftp(ssh, sftp_conn) + + +def _ensure_model_dir(sftp_conn, ssh, console: Console): + """Ensure the model directory exists on the device. + + Tries SFTP mkdir first. If that fails (permission denied in chroot), + falls back to SSH exec to create and chown the directory. + """ + try: + sftp_conn.stat(DEVICE_MODEL_DIR) + return + except FileNotFoundError: + pass + + console.print(f"[blue]Creating model directory: {DEVICE_MODEL_DIR}[/blue]") + + # Try SFTP mkdir first + try: + sftp_conn.mkdir(DEVICE_MODEL_DIR) + return + except Exception: + pass + + # SFTP failed — try SSH command to create with proper ownership + real_path = f"/opt/scifi/data{DEVICE_MODEL_DIR}" + try: + _, stdout, stderr = ssh.exec_command( + f"mkdir -p {real_path} && chown $(whoami) {real_path}" + ) + exit_code = stdout.channel.recv_exit_status() + if exit_code == 0: + console.print(f"[green]Created {DEVICE_MODEL_DIR}[/green]") + return + except Exception: + pass + + console.print( + f"[bold red]Error: Could not create model directory on device.[/bold red]" + ) + + +def _upload_file(sftp_conn, local_path: str, remote_path: str, console: Console): + """Upload a file to the device with progress display.""" + file_size = os.path.getsize(local_path) + + console.print(f"[blue]Uploading to {remote_path}...[/blue]") + + prog = progress.Progress( + progress.SpinnerColumn(), + progress.TextColumn("[progress.description]{task.description}"), + progress.BarColumn(), + progress.DownloadColumn(), + progress.TransferSpeedColumn(), + progress.TimeElapsedColumn(), + ) + + with prog: + task = prog.add_task("Uploading model", total=file_size) + + def update_progress(transferred: int, total: int): + prog.update(task, completed=transferred) + + sftp_conn.put(local_path, remote_path, callback=update_progress) + + console.print(f"[green]Uploaded to {remote_path}[/green]") diff --git a/synapse/cli/taps.py b/synapse/cli/taps.py index 0a697a3..27db77e 100644 --- a/synapse/cli/taps.py +++ b/synapse/cli/taps.py @@ -112,9 +112,9 @@ def list_taps(args): taps = tap.list_taps() table = Table(title="Available Taps", show_lines=True) - table.add_column("Name", style="cyan") - table.add_column("Message Type", style="green") - table.add_column("Endpoint", style="green") + table.add_column("Name", style="cyan", overflow="fold") + table.add_column("Message Type", style="green", overflow="fold") + table.add_column("Endpoint", style="green", overflow="fold") for tap in taps: table.add_row(tap.name, tap.message_type, tap.endpoint) diff --git a/synapse/client/sftp.py b/synapse/client/sftp.py index f8e5a72..c4d5525 100644 --- a/synapse/client/sftp.py +++ b/synapse/client/sftp.py @@ -1,4 +1,5 @@ import logging +import socket import paramiko import paramiko.ssh_exception @@ -27,20 +28,28 @@ def connect_sftp(hostname, username, password=None, pass_filename=None, key_file except Exception as e: logging.error(f"Failed to read password file: {e}") return None, None - try: + try: ssh.connect( hostname=hostname, port=port, username=username, password=password, key_filename=key_filename, - timeout=5 + timeout=10, + allow_agent=False, + look_for_keys=False, ) sftp = ssh.open_sftp() except TimeoutError as e: logging.error(f"Connection to {hostname} timed out") return None, None - + except socket.error as e: + logging.error(f"Socket error connecting to {hostname}:{port}: {e}") + return None, None + except paramiko.ssh_exception.SSHException as e: + logging.error(f"SSH error connecting to {hostname}:{port}: {e}") + raise # Re-raise to let caller handle it + return ssh, sftp def close_sftp(ssh, sftp): diff --git a/synapse/utils/model_converter/__init__.py b/synapse/utils/model_converter/__init__.py new file mode 100644 index 0000000..ac579f3 --- /dev/null +++ b/synapse/utils/model_converter/__init__.py @@ -0,0 +1,5 @@ +"""Model conversion utilities for deploying ML models to Synapse devices.""" + +from synapse.utils.model_converter.convert import convert_to_dlc + +__all__ = ["convert_to_dlc"] diff --git a/synapse/utils/model_converter/convert.py b/synapse/utils/model_converter/convert.py new file mode 100644 index 0000000..f21278c --- /dev/null +++ b/synapse/utils/model_converter/convert.py @@ -0,0 +1,88 @@ +"""Main model conversion pipeline.""" + +import os +import shutil +from typing import Optional + +from rich.console import Console + +from synapse.utils.model_converter.pt_to_onnx import convert_pt_to_onnx +from synapse.utils.model_converter.onnx_to_dlc import convert_onnx_to_dlc + + +def convert_to_dlc( + model_path: str, + input_shape: Optional[tuple[int, ...]] = None, + output_path: Optional[str] = None, + snpe_root: Optional[str] = None, + quantize: bool = False, + input_list: Optional[str] = None, + console: Optional[Console] = None, +) -> Optional[str]: + """Convert a model for deployment to Synapse devices. + + Handles .pt (PyTorch), .onnx, and .dlc files: + - .pt -> ONNX (on host) -> quantized DLC (in Docker) + - .onnx -> quantized DLC (in Docker) + - .dlc -> returns as-is + + Args: + model_path: Path to the model file (.pt, .onnx, or .dlc) + input_shape: Input shape for the model (required if model has dynamic dims) + output_path: Optional output path + snpe_root: Path to the QAIRT SDK + quantize: Whether to quantize the model to INT8 + input_list: Path to representative input list file (required if quantize=True) + console: Rich console for output + + Returns: + Path to the output DLC file, or None if conversion failed + """ + if not os.path.exists(model_path): + if console: + console.print( + f"[bold red]Error:[/bold red] Model file not found: {model_path}" + ) + return None + + ext = os.path.splitext(model_path)[1].lower() + + if ext == ".dlc": + if output_path and output_path != model_path: + shutil.copy2(model_path, output_path) + return output_path + return model_path + + kwargs = dict( + input_shape=input_shape, + output_path=output_path, + snpe_root=snpe_root, + quantize=quantize, + input_list=input_list, + console=console, + ) + + if ext == ".pt": + if console: + console.print("[bold blue]Step 1/2:[/bold blue] Converting PyTorch to ONNX...") + + onnx_path = convert_pt_to_onnx( + model_path, output_path=None, input_shape=input_shape, console=console, + ) + if onnx_path is None: + return None + + if console: + console.print("[bold blue]Step 2/2:[/bold blue] Converting ONNX (Docker)...") + + return convert_onnx_to_dlc(onnx_path, **kwargs) + + if ext == ".onnx": + if console: + console.print("[bold blue]Converting ONNX (Docker)...[/bold blue]") + return convert_onnx_to_dlc(model_path, **kwargs) + + if console: + console.print(f"[bold red]Error:[/bold red] Unsupported file type: {ext}") + console.print("[yellow]Supported formats: .pt, .onnx, .dlc[/yellow]") + return None diff --git a/synapse/utils/model_converter/docker/Dockerfile b/synapse/utils/model_converter/docker/Dockerfile new file mode 100644 index 0000000..04a870f --- /dev/null +++ b/synapse/utils/model_converter/docker/Dockerfile @@ -0,0 +1,17 @@ +FROM python:3.10-slim + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libc++1 \ + libatomic1 \ + && rm -rf /var/lib/apt/lists/* + +RUN pip install --no-cache-dir \ + "numpy<2" \ + "onnx==1.16.2" \ + pyyaml \ + packaging \ + "protobuf>=3.20,<5" + +COPY convert.py /opt/model-converter/convert.py + +ENTRYPOINT ["python3", "/opt/model-converter/convert.py"] diff --git a/synapse/utils/model_converter/docker/convert.py b/synapse/utils/model_converter/docker/convert.py new file mode 100644 index 0000000..d093395 --- /dev/null +++ b/synapse/utils/model_converter/docker/convert.py @@ -0,0 +1,264 @@ +#!/usr/bin/env python3 +"""ONNX to QNN context binary conversion script. + +Runs inside the synapse-model-converter Docker container. +Pipeline: ONNX → qairt-converter → DLC → qairt-quantizer → quantized DLC + +Expects: + - QAIRT SDK mounted at the path given by --snpe-root + - Input ONNX model accessible at --input + - Output directory writable at --output parent +""" + +import argparse +import os +import shutil +import subprocess +import sys +import tempfile + + +# --------------------------------------------------------------------------- +# ONNX helpers +# --------------------------------------------------------------------------- + +def get_input_shapes(onnx_path): + import onnx + + model = onnx.load(onnx_path) + inputs = [] + for inp in model.graph.input: + shape = [] + for dim in inp.type.tensor_type.shape.dim: + if dim.dim_param: + shape.append(dim.dim_param) + else: + shape.append(dim.dim_value) + inputs.append((inp.name, shape)) + return inputs + + +def has_dynamic_shapes(onnx_path): + for _, shape in get_input_shapes(onnx_path): + for dim in shape: + if isinstance(dim, str) or dim == 0: + return True + return False + + +# --------------------------------------------------------------------------- +# Tool finders +# --------------------------------------------------------------------------- + +def find_tool(snpe_root, name): + """Find a tool in the SDK bin directory.""" + path = os.path.join(snpe_root, "bin", "x86_64-linux-clang", name) + return path if os.path.exists(path) else None + + +def python_env(snpe_root): + """Environment for running Python-based SDK tools.""" + env = os.environ.copy() + env["SNPE_ROOT"] = snpe_root + env["PYTHONPATH"] = os.path.join(snpe_root, "lib", "python") + env["LD_LIBRARY_PATH"] = "/usr/local/lib:/usr/lib/x86_64-linux-gnu" + return env + + +def native_env(snpe_root): + """Environment for running native (C++) SDK tools.""" + env = os.environ.copy() + env["SNPE_ROOT"] = snpe_root + lib_dir = os.path.join(snpe_root, "lib", "x86_64-linux-clang") + env["LD_LIBRARY_PATH"] = f"{lib_dir}:/usr/local/lib:/usr/lib/x86_64-linux-gnu" + bin_dir = os.path.join(snpe_root, "bin", "x86_64-linux-clang") + env["PATH"] = f"{bin_dir}:{env.get('PATH', '')}" + return env + + +# --------------------------------------------------------------------------- +# Step 1: ONNX → DLC (qairt-converter) +# --------------------------------------------------------------------------- + +def convert_to_dlc(input_path, output_path, snpe_root, input_shape=None, input_name=None): + """Convert ONNX model to DLC using qairt-converter.""" + if has_dynamic_shapes(input_path): + if input_shape is None: + shapes = get_input_shapes(input_path) + print("ERROR: Model has dynamic input shapes.", file=sys.stderr) + print("Current input shapes:", file=sys.stderr) + for name, shape in shapes: + print(f" {name}: {shape}", file=sys.stderr) + print( + "\nPlease provide --input-shape with concrete dimensions.", + file=sys.stderr, + ) + return False + print(f"Using provided input shape {input_shape} for dynamic model") + + if input_name is None: + shapes = get_input_shapes(input_path) + input_name = shapes[0][0] if shapes else "input" + + # Try qairt-converter first (unified, preferred), fall back to snpe-onnx-to-dlc + converter = find_tool(snpe_root, "qairt-converter") + if converter: + cmd = [ + sys.executable, + converter, + "-i", input_path, + "-o", output_path, + ] + if input_shape is not None: + shape_str = ",".join(str(d) for d in input_shape) + cmd.extend(["-d", input_name, shape_str]) + else: + converter = find_tool(snpe_root, "snpe-onnx-to-dlc") + if converter is None: + print("ERROR: No converter found (tried qairt-converter, snpe-onnx-to-dlc)", + file=sys.stderr) + return False + cmd = [ + sys.executable, + converter, + "--input_network", input_path, + "--output_path", output_path, + ] + if input_shape is not None: + shape_str = ",".join(str(d) for d in input_shape) + cmd.extend(["-d", input_name, shape_str]) + + print(f"Converting ONNX to DLC: {' '.join(cmd)}") + result = subprocess.run(cmd, env=python_env(snpe_root), + capture_output=True, text=True, timeout=300) + + if result.returncode != 0: + print("ERROR: DLC conversion failed:", file=sys.stderr) + if result.stderr: + print(result.stderr, file=sys.stderr) + if result.stdout: + print(result.stdout) + return False + + if not os.path.exists(output_path): + print("ERROR: Converter ran but DLC file was not created", file=sys.stderr) + return False + + print(f"Successfully converted to {output_path}") + return True + + +# --------------------------------------------------------------------------- +# Step 2: Quantize DLC (qairt-quantizer) +# --------------------------------------------------------------------------- + +def quantize_dlc(dlc_path, input_list, snpe_root, output_path=None): + """Quantize a DLC model to INT8 using representative input data.""" + # Try qairt-quantizer first, fall back to snpe-dlc-quant + quantizer = find_tool(snpe_root, "qairt-quantizer") + if quantizer: + is_python = True + else: + quantizer = find_tool(snpe_root, "snpe-dlc-quant") + is_python = False + if quantizer is None: + print("ERROR: No quantizer found (tried qairt-quantizer, snpe-dlc-quant)", + file=sys.stderr) + return False + + if output_path is None: + base, ext = os.path.splitext(dlc_path) + output_path = f"{base}_quantized{ext}" + + if is_python: + cmd = [sys.executable, quantizer, "-i", dlc_path, "-l", input_list, + "-o", output_path] + env = python_env(snpe_root) + else: + cmd = [quantizer, "--input_dlc", dlc_path, "--input_list", input_list, + "--output_dlc", output_path] + env = native_env(snpe_root) + + # The quantizer resolves raw file paths relative to cwd. Create a temp + # working directory with symlinks so the data mount can stay read-only. + input_list_dir = os.path.dirname(os.path.abspath(input_list)) + work_dir = tempfile.mkdtemp() + for name in os.listdir(input_list_dir): + src = os.path.join(input_list_dir, name) + dst = os.path.join(work_dir, name) + os.symlink(src, dst) + + print(f"Quantizing model: {' '.join(cmd)}") + result = subprocess.run(cmd, env=env, capture_output=True, text=True, + timeout=600, cwd=work_dir) + + shutil.rmtree(work_dir, ignore_errors=True) + + if result.returncode != 0: + print("ERROR: Quantization failed:", file=sys.stderr) + if result.stderr: + print(result.stderr, file=sys.stderr) + if result.stdout: + print(result.stdout) + return False + + if not os.path.exists(output_path): + print("ERROR: Quantizer ran but output file not created", file=sys.stderr) + return False + + # Replace the float DLC with the quantized one + shutil.move(output_path, dlc_path) + print(f"Successfully quantized model to {dlc_path}") + return True + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + +def main(): + parser = argparse.ArgumentParser( + description="Convert ONNX model to quantized DLC for Qualcomm HTP inference" + ) + parser.add_argument("--input", required=True, help="Path to input ONNX model") + parser.add_argument("--output", required=True, help="Path for output file") + parser.add_argument( + "--snpe-root", required=True, help="Path to QAIRT SDK root" + ) + parser.add_argument( + "--input-shape", default=None, help="Input shape (comma-separated, e.g. 1,1920)" + ) + parser.add_argument("--input-name", default=None, help="Input tensor name") + parser.add_argument( + "--quantize", action="store_true", help="Quantize model to INT8" + ) + parser.add_argument( + "--input-list", default=None, help="Input list file for quantization" + ) + args = parser.parse_args() + + input_shape = None + if args.input_shape: + input_shape = tuple(int(x.strip()) for x in args.input_shape.split(",")) + + # Output is always DLC + dlc_path = args.output + + # Step 1: Convert ONNX → DLC + success = convert_to_dlc( + args.input, dlc_path, args.snpe_root, + input_shape=input_shape, input_name=args.input_name, + ) + + # Step 2: Quantize (required for DSP inference) + if success and args.quantize: + if not args.input_list: + print("ERROR: --quantize requires --input-list", file=sys.stderr) + sys.exit(1) + success = quantize_dlc(dlc_path, args.input_list, args.snpe_root) + + sys.exit(0 if success else 1) + + +if __name__ == "__main__": + main() diff --git a/synapse/utils/model_converter/onnx_to_dlc.py b/synapse/utils/model_converter/onnx_to_dlc.py new file mode 100644 index 0000000..5b24f99 --- /dev/null +++ b/synapse/utils/model_converter/onnx_to_dlc.py @@ -0,0 +1,252 @@ +"""ONNX to DLC conversion via Docker container. + +The conversion runs inside a Docker container that has Python 3.10 and +the required dependencies pre-installed. The user's SNPE/QAIRT SDK is +bind-mounted at runtime (not baked into the image) to comply with +Qualcomm's license terms. +""" + +import os +import shutil +import subprocess +import tempfile +from typing import Optional + +from rich.console import Console + +DOCKER_IMAGE = "synapse-model-converter:latest" + + +def _find_model_converter_dir() -> str: + """Locate the directory containing the Dockerfile for model conversion.""" + here = os.path.dirname(os.path.abspath(__file__)) + docker_dir = os.path.join(here, "docker") + if os.path.isdir(docker_dir) and os.path.isfile( + os.path.join(docker_dir, "Dockerfile") + ): + return docker_dir + + raise FileNotFoundError( + "Model converter Dockerfile not found. " + "Try reinstalling: pip install science-synapse" + ) + + +def _image_exists() -> bool: + """Check if the Docker image is already built.""" + result = subprocess.run( + ["docker", "image", "inspect", DOCKER_IMAGE], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + return result.returncode == 0 + + +def _build_image(console: Optional[Console] = None) -> bool: + """Build the model-converter Docker image.""" + try: + build_dir = _find_model_converter_dir() + except FileNotFoundError as e: + if console: + console.print(f"[bold red]Error:[/bold red] {e}") + return False + + if console: + console.print( + f"[yellow]Building Docker image [bold]{DOCKER_IMAGE}[/bold] " + f"(first time only)...[/yellow]" + ) + + try: + subprocess.run( + ["docker", "build", "-t", DOCKER_IMAGE, "."], + cwd=build_dir, + check=True, + ) + except subprocess.CalledProcessError: + if console: + console.print( + "[bold red]Error:[/bold red] Failed to build model-converter Docker image" + ) + return False + + if console: + console.print(f"[green]Docker image built successfully[/green]") + return True + + +def ensure_docker(console: Optional[Console] = None) -> bool: + """Check that Docker is available and the image is built.""" + if shutil.which("docker") is None: + if console: + console.print( + "[bold red]Error:[/bold red] Docker is required for model conversion " + "but was not found. Please install Docker." + ) + return False + + try: + subprocess.run( + ["docker", "info"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=True, + ) + except subprocess.CalledProcessError: + if console: + console.print( + "[bold red]Error:[/bold red] Docker daemon is not running. " + "Please start Docker and try again." + ) + return False + + if not _image_exists(): + return _build_image(console) + + return True + + +def convert_onnx_to_dlc( + onnx_path: str, + output_path: Optional[str] = None, + input_shape: Optional[tuple[int, ...]] = None, + input_name: str = "input", + snpe_root: Optional[str] = None, + quantize: bool = False, + input_list: Optional[str] = None, + console: Optional[Console] = None, +) -> Optional[str]: + """Convert an ONNX model to DLC using Docker. + + Args: + onnx_path: Path to the ONNX model + output_path: Optional output path for the output file + input_shape: Input shape (required if model has dynamic dims) + input_name: Name of the input tensor + snpe_root: Path to the QAIRT SDK + quantize: Whether to quantize the model to INT8 + input_list: Path to representative input list file for quantization + console: Rich console for output + + Returns: + Path to the output DLC file, or None on failure + """ + if snpe_root is None: + snpe_root = os.environ.get("SNPE_ROOT") or os.environ.get("QAIRT_ROOT") + + if snpe_root is None: + if console: + console.print( + "[bold red]Error:[/bold red] --snpe-root is required " + "(or set SNPE_ROOT / QAIRT_ROOT env var)" + ) + return None + + snpe_root = os.path.abspath(snpe_root) + if not os.path.isdir(snpe_root): + if console: + console.print( + f"[bold red]Error:[/bold red] SNPE root not found: {snpe_root}" + ) + return None + + if not ensure_docker(console): + return None + + # Resolve paths for Docker mounts + onnx_path = os.path.abspath(onnx_path) + onnx_dir = os.path.dirname(onnx_path) + onnx_filename = os.path.basename(onnx_path) + + if output_path is None: + base_name = os.path.splitext(onnx_filename)[0] + output_path = os.path.join(tempfile.gettempdir(), f"{base_name}.dlc") + + output_dir = os.path.abspath(os.path.dirname(output_path)) + output_filename = os.path.basename(output_path) + os.makedirs(output_dir, exist_ok=True) + + # Build docker run command + # Run as the host user so output files have correct ownership + cmd = [ + "docker", + "run", + "--rm", + "--user", + f"{os.getuid()}:{os.getgid()}", + "-v", + f"{onnx_dir}:/input:ro", + "-v", + f"{snpe_root}:/snpe:ro", + "-v", + f"{output_dir}:/output", + ] + + # Mount input data directory for quantization + if quantize and input_list: + input_list = os.path.abspath(input_list) + input_list_dir = os.path.dirname(input_list) + input_list_filename = os.path.basename(input_list) + cmd.extend(["-v", f"{input_list_dir}:/data:ro"]) + + cmd.extend([ + DOCKER_IMAGE, + "--input", + f"/input/{onnx_filename}", + "--output", + f"/output/{output_filename}", + "--snpe-root", + "/snpe", + ]) + + if input_shape is not None: + shape_str = ",".join(str(d) for d in input_shape) + cmd.extend(["--input-shape", shape_str]) + + if input_name != "input": + cmd.extend(["--input-name", input_name]) + + if quantize and input_list: + cmd.extend(["--quantize", "--input-list", f"/data/{input_list_filename}"]) + + if console: + console.print("[dim]Running conversion in Docker container...[/dim]") + + try: + result = subprocess.run( + cmd, capture_output=True, text=True, timeout=600, + ) + except subprocess.TimeoutExpired: + if console: + console.print( + "[bold red]Error:[/bold red] Conversion timed out after 10 minutes" + ) + return None + + # Display container output + if result.stdout: + for line in result.stdout.strip().split("\n"): + if line.strip(): + if console: + console.print(f" {line}") + + if result.returncode != 0: + if console: + console.print("[bold red]DLC conversion failed:[/bold red]") + if result.stderr: + for line in result.stderr.strip().split("\n")[-15:]: + if line.strip(): + console.print(f"[red] {line}[/red]") + return None + + if not os.path.exists(output_path): + if console: + console.print( + "[bold red]Error:[/bold red] Container exited OK but DLC file not found" + ) + return None + + if console: + console.print(f"[green]Successfully converted to {output_path}[/green]") + + return output_path diff --git a/synapse/utils/model_converter/pt_to_onnx.py b/synapse/utils/model_converter/pt_to_onnx.py new file mode 100644 index 0000000..c7340eb --- /dev/null +++ b/synapse/utils/model_converter/pt_to_onnx.py @@ -0,0 +1,139 @@ +"""PyTorch to ONNX model conversion.""" + +import os +import tempfile +from typing import Optional + +from rich.console import Console + + +def convert_pt_to_onnx( + pt_path: str, + output_path: Optional[str] = None, + input_shape: Optional[tuple[int, ...]] = None, + console: Optional[Console] = None, +) -> Optional[str]: + """ + Convert a PyTorch model to ONNX format. + + Args: + pt_path: Path to the .pt file + output_path: Optional output path for the ONNX file. If None, uses temp directory. + input_shape: Input shape for the model (required for tracing) + console: Rich console for output + + Returns: + Path to the converted ONNX file, or None if conversion failed + """ + try: + import torch + except ImportError: + if console: + console.print("[bold red]Error:[/bold red] torch is required for PT to ONNX conversion") + console.print("[yellow]Install with: pip install torch[/yellow]") + return None + + if console: + console.print(f"[blue]Loading PyTorch model from {pt_path}...[/blue]") + + try: + model = torch.load(pt_path, map_location="cpu", weights_only=False) + except Exception as e: + if console: + console.print(f"[bold red]Failed to load PyTorch model:[/bold red] {e}") + return None + + # Handle case where saved file is a state_dict instead of a full model + if isinstance(model, dict): + if console: + console.print( + "[bold red]Error:[/bold red] The .pt file contains a state_dict, not a full model." + ) + console.print( + "[yellow]Hint: Save the model with torch.save(model, path) instead of " + "torch.save(model.state_dict(), path)[/yellow]" + ) + return None + + model.eval() + + # Determine input shape + if input_shape is None: + # Try to infer input shape from the model + input_shape = _infer_input_shape(model) + if input_shape is None: + if console: + console.print( + "[bold red]Error:[/bold red] Could not infer input shape. " + "Please provide --input-shape" + ) + return None + if console: + console.print(f"[green]Inferred input shape: {input_shape}[/green]") + + # Create dummy input + dummy_input = torch.randn(*input_shape) + + # Determine output path + if output_path is None: + base_name = os.path.splitext(os.path.basename(pt_path))[0] + output_path = os.path.join(tempfile.gettempdir(), f"{base_name}.onnx") + + if console: + console.print(f"[blue]Exporting to ONNX: {output_path}...[/blue]") + + try: + torch.onnx.export( + model, + dummy_input, + output_path, + export_params=True, + opset_version=13, + do_constant_folding=True, + input_names=["input"], + output_names=["output"], + dynamic_axes=None, # Static shapes for device deployment + ) + except Exception as e: + if console: + console.print(f"[bold red]ONNX export failed:[/bold red] {e}") + return None + + if console: + console.print(f"[green]Successfully exported to {output_path}[/green]") + + return output_path + + +def _infer_input_shape(model) -> Optional[tuple[int, ...]]: + """ + Try to infer input shape from model structure. + + Returns: + Inferred input shape tuple, or None if cannot be inferred + """ + try: + import torch.nn as nn + + # Check if it's a Sequential model with a first Linear layer + if isinstance(model, nn.Sequential): + first_layer = list(model.children())[0] + if isinstance(first_layer, nn.Linear): + return (1, first_layer.in_features) + + # Check for common first layer patterns + for name, module in model.named_modules(): + if isinstance(module, nn.Linear) and "." not in name: + return (1, module.in_features) + if isinstance(module, nn.Conv1d) and "." not in name: + # For Conv1d, we need channels and sequence length + # Use a reasonable default sequence length + return (1, module.in_channels, 64) + if isinstance(module, nn.Conv2d) and "." not in name: + # For Conv2d, use a reasonable default spatial size + return (1, module.in_channels, 32, 32) + + except Exception: + pass + + return None