Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion application/single_app/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@
EXECUTOR_TYPE = 'thread'
EXECUTOR_MAX_WORKERS = 30
SESSION_TYPE = 'filesystem'
VERSION = "0.250.065"
VERSION = "0.250.066"
IS_DEVELOPMENT = is_development_env_enabled()

SESSION_COOKIE_SAMESITE = os.getenv('SESSION_COOKIE_SAMESITE', 'Lax')
Expand Down
53 changes: 53 additions & 0 deletions application/single_app/functions_model_capabilities.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# functions_model_capabilities.py

import re
from collections.abc import Mapping


MODEL_IDENTIFIER_SEPARATOR_PATTERN = re.compile(r"[\s_.]+")

Check warning on line 7 in application/single_app/functions_model_capabilities.py

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Moderate - Changed line contains obfuscation, dynamic loading, or hidden payload marker. Recommendation%3A Confirm the changed code is not hiding behavior, decoding payloads, or bypassing normal review.
GPT_VISION_MODEL_PATTERN = re.compile(r"(?:^|-)gpt-(?:[5-9]|\d{2,})(?:-|$)")

Check warning on line 8 in application/single_app/functions_model_capabilities.py

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Moderate - Changed line contains obfuscation, dynamic loading, or hidden payload marker. Recommendation%3A Confirm the changed code is not hiding behavior, decoding payloads, or bypassing normal review.
O_SERIES_MODEL_PATTERN = re.compile(r"(?:^|-)o\d+(?:-|$)")

Check warning on line 9 in application/single_app/functions_model_capabilities.py

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Moderate - Changed line contains obfuscation, dynamic loading, or hidden payload marker. Recommendation%3A Confirm the changed code is not hiding behavior, decoding payloads, or bypassing normal review.
MODEL_IDENTIFIER_FIELDS = (
"modelName",
"displayName",
"deploymentName",
"deployment",
"name",
)


def _normalize_model_identifier(value):
return MODEL_IDENTIFIER_SEPARATOR_PATTERN.sub(
"-",
str(value or "").strip().lower(),
)


def is_vision_capable_model_name(*model_names):
"""Return whether any supplied identifier names a supported vision model."""
for model_name in model_names:
normalized_name = _normalize_model_identifier(model_name)
if (
"vision" in normalized_name
or "gpt-4o" in normalized_name
or "gpt-4-1" in normalized_name
or "gpt-4-5" in normalized_name
or GPT_VISION_MODEL_PATTERN.search(normalized_name)

Check warning on line 35 in application/single_app/functions_model_capabilities.py

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Important - Changed line contains secret or sensitive data source marker. Recommendation%3A Pair this source with any nearby network, logging, serialization, or process execution sink before approving.
or O_SERIES_MODEL_PATTERN.search(normalized_name)

Check warning on line 36 in application/single_app/functions_model_capabilities.py

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Important - Changed line contains secret or sensitive data source marker. Recommendation%3A Pair this source with any nearby network, logging, serialization, or process execution sink before approving.
):
return True

return False


def is_vision_capable_model(model):
"""Return whether a model record or identifier names a supported vision model."""
if isinstance(model, str):
return is_vision_capable_model_name(model)

if isinstance(model, Mapping):
model_names = [model.get(field_name) for field_name in MODEL_IDENTIFIER_FIELDS]
else:
model_names = [getattr(model, field_name, None) for field_name in MODEL_IDENTIFIER_FIELDS]

Check warning on line 51 in application/single_app/functions_model_capabilities.py

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Moderate - Changed line contains obfuscation, dynamic loading, or hidden payload marker. Recommendation%3A Confirm the changed code is not hiding behavior, decoding payloads, or bypassing normal review.

return is_vision_capable_model_name(*model_names)
4 changes: 3 additions & 1 deletion application/single_app/route_frontend_admin_settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
from functions_notifications import broadcast_system_notification
from functions_logging import *
from functions_document_actions import normalize_document_action_capabilities
from functions_model_capabilities import is_vision_capable_model
from functions_terms_of_use import (
TERMS_OF_USE_DEFAULT_REDIRECT,
TERMS_OF_USE_MAX_BUTTON_TEXT_LENGTH,
Expand Down Expand Up @@ -596,7 +597,8 @@ def admin_settings():
chunk_size_cap=get_chunk_size_cap(settings),
chunk_size_effective=get_chunk_size_config(settings),
audio_runtime_capabilities=audio_runtime_capabilities,
source_review_runtime_capabilities=source_review_runtime_capabilities
source_review_runtime_capabilities=source_review_runtime_capabilities,
is_vision_capable_model=is_vision_capable_model,
# You don't need to pass deployments separately if they are added to settings['..._model']['all']
# gpt_deployments=gpt_deployments,
# embedding_deployments=embedding_deployments,
Expand Down
40 changes: 26 additions & 14 deletions application/single_app/static/js/admin/admin_settings.js
Original file line number Diff line number Diff line change
Expand Up @@ -8364,18 +8364,22 @@ const visionToggle = document.getElementById('enable_multimodal_vision');
const visionModelDiv = document.getElementById('multimodal_vision_model_settings');
const visionSelect = document.getElementById('multimodal_vision_model');

function isVisionCapableModelName(modelName) {
const modelNameLower = (modelName || '').toLowerCase();
return (
modelNameLower.includes('vision') ||
modelNameLower.includes('gpt-4o') ||
modelNameLower.includes('gpt-4.1') ||
modelNameLower.includes('gpt-4.5') ||
modelNameLower.includes('gpt-5') ||
/^o\d+/.test(modelNameLower) ||
modelNameLower.includes('o1-') ||
modelNameLower.includes('o3-')
);
function isVisionCapableModelName(...modelNames) {
return modelNames.some(modelName => {
const normalizedName = String(modelName || '')
.trim()
.toLowerCase()
.replace(/[\s_.]+/g, '-');

return (
normalizedName.includes('vision') ||
normalizedName.includes('gpt-4o') ||
normalizedName.includes('gpt-4-1') ||
normalizedName.includes('gpt-4-5') ||
/(?:^|-)gpt-(?:[5-9]|\d{2,})(?:-|$)/.test(normalizedName) ||
/(?:^|-)o\d+(?:-|$)/.test(normalizedName)
);
});
}

function getSelectedVisionModelOption() {
Expand All @@ -8401,10 +8405,18 @@ function populateVisionModels() {
.filter(ep => ep && ep.enabled)
.forEach(ep => {
(ep.models || [])
.filter(m => m && m.enabled && isVisionCapableModelName(m.modelName || m.displayName))
.filter(m => m && m.enabled && isVisionCapableModelName(
m.modelName,
m.displayName,
m.deploymentName,
m.deployment,
m.name
))
.forEach(m => {
const value = m.deploymentName;
const label = `${m.displayName || m.deploymentName} (${m.modelName})`;
const label = m.modelName
? `${m.displayName || m.deploymentName} (${m.modelName})`
: (m.displayName || m.deploymentName);
const opt = new Option(label, value);
opt.dataset.endpointId = ep.id || '';
opt.dataset.modelId = m.id || '';
Expand Down
48 changes: 6 additions & 42 deletions application/single_app/templates/admin_settings.html
Original file line number Diff line number Diff line change
Expand Up @@ -8213,27 +8213,15 @@ <h5>
{# Iterate all enabled endpoints and models, filter to vision-capable #}
{% for endpoint in settings.model_endpoints if endpoint.enabled %}
{% for m in endpoint.models if m.enabled %}
{% set model_lower = (m.modelName or m.displayName or '').lower() %}
{% set is_vision =
'vision' in model_lower or
'gpt-4o' in model_lower or
'gpt-4.1' in model_lower or
'gpt-4.5' in model_lower or
'-5' in model_lower or
model_lower.startswith('o1') or
model_lower.startswith('o3') or
'o1-' in model_lower or
'o3-' in model_lower
%}
{% if is_vision %}
{% if is_vision_capable_model is defined and is_vision_capable_model(m) %}
{% set option_value = m.deploymentName %}
<option value="{{ option_value }}"
data-endpoint-id="{{ endpoint.id or '' }}"
data-model-id="{{ m.id or '' }}"
data-provider="{{ endpoint.provider or '' }}"
data-model-name="{{ m.modelName or '' }}"
{% if settings.multimodal_vision_model == option_value %}selected{% endif %}>
{{ m.displayName or m.deploymentName }} ({{ m.modelName }})
{{ m.displayName or m.deploymentName }}{% if m.modelName %} ({{ m.modelName }}){% endif %}
</option>
{% endif %}
{% endfor %}
Expand All @@ -8242,19 +8230,7 @@ <h5>
{# Legacy APIM-based GPT configuration #}
{% for d in (settings.azure_apim_gpt_deployment or '').split(',') if d %}
{% set d_trim = d.strip() %}
{% set model_lower = d_trim.lower() %}
{% set is_vision =
'vision' in model_lower or
'gpt-4o' in model_lower or
'gpt-4.1' in model_lower or
'gpt-4.5' in model_lower or
'-5' in model_lower or
model_lower.startswith('o1') or
model_lower.startswith('o3') or
'o1-' in model_lower or
'o3-' in model_lower
%}
{% if is_vision %}
{% if is_vision_capable_model is defined and is_vision_capable_model(d_trim) %}
<option value="{{ d_trim }}"
{% if settings.multimodal_vision_model == d_trim %}selected{% endif %}>
{{ d_trim }}
Expand All @@ -8264,28 +8240,16 @@ <h5>
{% else %}
{# Legacy single-endpoint GPT configuration #}
{% for m in settings.gpt_model.selected %}
{% set model_lower = (m.modelName or '').lower() %}
{% set is_vision =
'vision' in model_lower or
'gpt-4o' in model_lower or
'gpt-4.1' in model_lower or
'gpt-4.5' in model_lower or
'gpt-5' in model_lower or
model_lower.startswith('o1') or
model_lower.startswith('o3') or
'o1-' in model_lower or
'o3-' in model_lower
%}
{% if is_vision %}
{% if is_vision_capable_model is defined and is_vision_capable_model(m) %}
<option value="{{ m.deploymentName }}"
{% if settings.multimodal_vision_model == m.deploymentName %}selected{% endif %}>
{{ m.deploymentName }} ({{ m.modelName }})
{{ m.deploymentName }}{% if m.modelName %} ({{ m.modelName }}){% endif %}
</option>
{% endif %}
{% endfor %}
{% endif %}
</select>
<div class="form-text">Select a GPT model with vision capabilities (e.g., gpt-4o, gpt-4-vision, gpt-5, gpt-5-nano, etc.). Only vision-capable models are shown.</div>
<div class="form-text">Select a GPT model with vision capabilities (for example, gpt-4o or supported GPT 5 and later models). Only vision-capable models are shown.</div>

<button type="button" class="btn btn-secondary mt-3" id="test_multimodal_vision_button">
<i class="bi bi-clipboard-check me-1"></i>Test Vision Analysis
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Multi-Modal Vision Analysis Feature

**Version**: 0.229.088 (Initial), 0.229.089 (Expanded model support)
**Version**: 0.229.088 (Initial), 0.229.089 (Expanded model support), 0.250.066 (GPT 5.6+ endpoint aliases)
**Implemented**: November 21, 2025

## Overview
Expand Down Expand Up @@ -337,8 +337,8 @@ group-documents/
- `o3` - Next-generation reasoning model (if available)
- `o3-mini` - Compact version (if available)

**GPT-5 Series**:
- All GPT-5 models support vision (when available in your region)
**GPT-5 and Later Series**:
- GPT 5.6 and later GPT model variants are available when they support vision in your region

**GPT-4.5 & GPT-4.1 Series**:
- `gpt-4.5` - Mid-generation update with vision
Expand All @@ -350,7 +350,7 @@ group-documents/
- `gpt-4-vision` - Standard vision variant

**Model Detection**:
The admin UI automatically filters and displays only vision-capable models based on naming patterns. If your deployed model includes "vision", "gpt-4o", "gpt-4.1", "gpt-4.5", "gpt-5", or matches o-series patterns (o1, o3, etc.), it will appear in the dropdown.
The admin UI filters vision-capable models using the model name, display name, and deployment name supplied by each enabled endpoint. Matching is case-insensitive and treats spaces, periods, and underscores as separators. Models identified as "vision", "gpt-4o", "gpt-4.1", "gpt-4.5", GPT major version 5 or later, or supported o-series patterns (o1, o3, etc.) appear in the dropdown.

**API Requirements**:
- Azure OpenAI API version 2024-02-15-preview or later
Expand Down Expand Up @@ -494,15 +494,15 @@ print(doc.get('vision_analysis'))
- Contains "gpt-4o" (gpt-4o, gpt-4o-mini)
- Contains "gpt-4.1" (any GPT-4.1 variant)
- Contains "gpt-4.5" (any GPT-4.5 variant)
- Contains "gpt-5" (any GPT-5 variant)
- Identifies GPT major version 5 or later in its model, display, or deployment name
- Matches o-series pattern (o1, o1-preview, o1-mini, o3, o3-mini)
2. Was model fetched and selected in Chat Model?
3. For APIM: Is deployment name listed in comma-separated list?

**Supported Model Families**:
- **GPT-4o series**: gpt-4o, gpt-4o-mini
- **O-series**: o1, o1-preview, o1-mini, o3, o3-mini
- **GPT-5 series**: All variants (when available)
- **GPT-5 and later series**: GPT 5.6 and later variants (when available)
- **GPT-4.5 & GPT-4.1**: All variants
- **Legacy vision**: gpt-4-vision, gpt-4-turbo-vision

Expand All @@ -516,7 +516,7 @@ print(doc.get('vision_analysis'))
6. Model should now appear in dropdown
```

**Note**: If your model supports vision but doesn't appear, check that the deployment name follows Azure OpenAI naming conventions. Custom deployment names may not be detected automatically if they don't include the model family identifiers.
**Note**: If a custom deployment alias does not identify the model family, ensure its model name or display name contains the supported family identifier.

## Related Features

Expand Down
61 changes: 61 additions & 0 deletions docs/explanation/fixes/GPT_5_6_MULTIMODAL_VISION_SELECTOR_FIX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# GPT 5.6+ Multi-Modal Vision Selector Fix (v0.250.066)

Fixed in version: **0.250.066**

Related issue: [#1086](https://github.com/microsoft/simplechat/issues/1086)

Check warning on line 5 in docs/explanation/fixes/GPT_5_6_MULTIMODAL_VISION_SELECTOR_FIX.md

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Important - Changed line contains external connection or remote asset marker. Recommendation%3A Review whether changed code can send prompts, files, credentials, cookies, settings, logs, or user data to a new sink.

## Issue

Enabled GPT 5.6 deployments appeared in the Global Endpoints table but were
missing from the Multi-Modal Vision Analysis model selector. The selector could
therefore leave admins on GPT 5.4 even when Luna, Sol, Terra, or later GPT
deployments were configured.

## Root Cause

The browser-side vision filter inspected only `modelName` or `displayName` and
required narrow punctuation patterns. Endpoint records that exposed the model
family through `deploymentName`, used a provider prefix, or formatted a display
name as `GPT 5.6` did not match.

## Technical Details

### Files Modified

- `application/single_app/static/js/admin/admin_settings.js`
- `application/single_app/functions_model_capabilities.py`
- `application/single_app/route_frontend_admin_settings.py`
- `application/single_app/templates/admin_settings.html`
- `application/single_app/config.py`
- `ui_tests/test_admin_multimodal_vision_model_options.py`
- `docs/explanation/features/v0.229.088/MULTIMODAL_VISION_ANALYSIS.md`

### Code Changes

- Normalized spaces, periods, and underscores before capability matching.
- Recognized GPT major version 5 and later, including provider-prefixed names.
- Evaluated model, display, deployment, and fallback name fields.
- Applied the same capability contract during initial server rendering and
client-side endpoint updates.
- Allowed templates loaded ahead of a restarted application worker to fall
back to client-side option population instead of returning a Jinja error.
- Preserved filtering for disabled endpoints, disabled models, and unsupported

Check warning on line 42 in docs/explanation/fixes/GPT_5_6_MULTIMODAL_VISION_SELECTOR_FIX.md

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Important - Changed line contains security control, sanitization, or audit marker. Recommendation%3A Confirm the change does not weaken auth, CSRF, CSP, XSS defenses, settings sanitization, redaction, audit logging, or tests.
model families.
- Incremented `config.py` from `0.250.065` to `0.250.066`.

## Impact

Azure OpenAI, New Foundry, and classic Foundry endpoints can expose supported

Check warning on line 48 in docs/explanation/fixes/GPT_5_6_MULTIMODAL_VISION_SELECTOR_FIX.md

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Important - Changed line contains secret or sensitive data source marker. Recommendation%3A Pair this source with any nearby network, logging, serialization, or process execution sink before approving.
GPT 5.6 and later deployments in the same Vision Model selector without
requiring one exact metadata field or separator format.

## Validation

The focused Playwright regression test executes the production matcher, actual
Jinja selector, and dropdown population functions with GPT 5.6 Luna, Sol,
Terra, GPT 5.7, GPT-4o, disabled model, disabled endpoint, and non-GPT cases.

Check warning on line 56 in docs/explanation/fixes/GPT_5_6_MULTIMODAL_VISION_SELECTOR_FIX.md

View workflow job for this annotation

GitHub Actions / malicious-pr-security-review

Important - Changed line contains security control, sanitization, or audit marker. Recommendation%3A Confirm the change does not weaken auth, CSRF, CSP, XSS defenses, settings sanitization, redaction, audit logging, or tests.
It also renders the selector without the route helper to verify that a stale
worker cannot return an `UndefinedError`.

Before the fix, only the placeholder and GPT-4o option were rendered. After the
fix, all enabled supported options render while excluded models remain absent.
9 changes: 9 additions & 0 deletions docs/explanation/release_notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

For feature-focused and fix-focused drill-downs by version, see [Features by Version](/explanation/features/) and [Fixes by Version](/explanation/fixes/).

### **(v0.250.066)**

#### Bug Fixes

* **GPT 5.6+ Multi-Modal Vision Model Selection**
* Enabled GPT 5.6 Luna, Sol, Terra, and later supported GPT deployments to appear in the Multi-Modal Vision Analysis selector across Azure OpenAI and Foundry endpoints.
* Model detection now evaluates model, display, and deployment names with normalized separators while preserving disabled-model and unsupported-family filtering.
* (Ref: microsoft/simplechat#1086, `admin_settings.js`, `test_admin_multimodal_vision_model_options.py`)

### **(v0.250.065)**

#### New Features
Expand Down
Loading
Loading