Implement a AI prompt refactoring method

[janwiemers]

As a user I want to get feedback on my prompt as a standalone idea but also in terms of the overall flow.

the idea would be that we have a bedrock integration that analyses the current workflow and provides valuable feedback to the user including suggested changes.

[janwiemers]

Feature Specification: AI-Powered Instruction Optimization

Version: 1.0
Date: 2025-12-01
Status: Draft

Overview

Add AI-powered optimization features to help users write better test instructions and debug failed test executions using Amazon Bedrock.

---

Feature 1: Instruction Optimizer (Iteration 1)

User Story

As a test creator, I want to optimize my natural language instructions so that Nova Act can execute them more reliably.

Requirements

• Add "Optimize" button next to the instructions field in the shared form
• On click, send instruction text to Amazon Bedrock with optimization prompt
• Display optimized instruction for user review
• Allow user to accept/reject/edit the optimized version

UI Components

Button: "Optimize Instruction" (icon: sparkle/wand)

• Placement: Next to instructions field in step form
• States: Default, Loading, Disabled

Results Display:

• Side-by-side comparison or modal showing original vs optimized
• Actions: "Apply", "Cancel", "Edit & Apply"

API Specification

Endpoint:

POST /usecase/{id}/steps/{stepId}/optimize

Request Body:

{
  "instruction": "string"
}

Response:

{
  "original": "string",
  "optimized": "string",
  "suggestions": ["string"]
}

Status Codes:

• 200: Success
• 400: Invalid instruction
• 429: Rate limit exceeded
• 500: Bedrock service error

Bedrock Integration

Model: Claude 3.5 Sonnet

Prompt Template:

You are an expert at writing UI test instructions for AI automation.
Optimize the following test instruction to be clearer, more specific, and easier for an AI agent to execute.

Original instruction:
{user_instruction}

Provide an optimized version that:
- Uses clear, unambiguous language
- Specifies exact UI elements (buttons, fields, labels)
- Includes explicit wait conditions if needed
- Breaks complex actions into discrete steps
- Specifies expected outcomes

Return only the optimized instruction.

Implementation Considerations

Security:

• Sanitize user input before sending to Bedrock
• Don't send sensitive data (credentials, PII) in prompts
• Implement rate limiting (e.g., 10 requests per minute per user)

Performance:

• Cache optimization results for identical instructions
• Timeout: 30 seconds
• Show loading indicator during processing

Cost Management:

• Track token usage per optimization
• Consider usage limits per user/organization

Error Handling:

• Graceful degradation if Bedrock is unavailable
• User-friendly error messages
• Fallback: Allow manual editing

---

Feature 2: Failed Execution Analyzer (Iteration 2)

User Story

As a test creator, when my test fails, I want AI to analyze the failure and suggest fixes so I can quickly resolve issues.

Requirements

• Trigger available when test execution status = "failed"
• Send execution log (steps, actions, error messages, screenshots if available) to Bedrock
• Receive actionable suggestions: reordered steps, modified instructions, new steps, or alternative approaches
• Display suggestions in a structured format
• Allow user to apply suggestions individually or in combination

UI Components

Trigger: "Analyze Failure" button

• Placement: Failed test execution detail page
• States: Default, Loading, Disabled

Analysis Results Panel:

• Root cause summary
• Suggested fixes (categorized by type)
• Confidence level per suggestion
• Preview of changes
• Bulk apply or individual selection

API Specification

Endpoint:

POST /usecase/{id}/executions/{executionId}/analyze-failure

Request Body: None (execution details fetched server-side)

Response:

{
  "rootCause": "string",
  "suggestions": [
    {
      "type": "reorder" | "modify" | "add" | "remove",
      "stepIndex": "number",
      "original": "string",
      "proposed": "string",
      "reasoning": "string",
      "confidence": "high" | "medium" | "low"
    }
  ]
}

Status Codes:

• 200: Success
• 404: Execution not found
• 400: Execution not in failed state
• 429: Rate limit exceeded
• 500: Bedrock service error

Bedrock Integration

Model: Claude 3.5 Sonnet

Prompt Template:

You are an expert at debugging UI test automation failures.

Analyze this failed test execution and provide specific suggestions to fix it.

Test Instructions:
{original_instructions}

Execution Log:
{execution_steps}

Error:
{error_message}

Analyze the failure and provide:
1. Root cause of the failure
2. Specific suggestions to fix the test, including:
   - Steps that should be reordered (with new order)
   - Instructions that should be modified (with improved wording)
   - New steps that should be added (with exact placement)
   - Steps that should be removed

Format each suggestion with:
- Type of change
- Which step (by index or description)
- Proposed change
- Reasoning
- Confidence level

Be specific and actionable.

Implementation Considerations

Security:

• Sanitize execution logs before sending to Bedrock
• Redact sensitive data from logs
• Implement rate limiting (e.g., 5 requests per minute per user)

Performance:

• Async processing (can take 30-60 seconds)
• Show progress indicator
• Consider background job for long analyses
• Timeout: 60 seconds

Cost Management:

• Track token usage per analysis
• Consider usage limits per user/organization
• Use streaming responses for better UX

Error Handling:

• Graceful degradation if Bedrock is unavailable
• Timeout handling
• User-friendly error messages

---

Data Model Extensions

Instruction History

Track optimization history for analytics and rollback.

{
  "instructionId": "uuid",
  "testCaseId": "uuid",
  "stepId": "uuid",
  "originalText": "string",
  "optimizedText": "string",
  "appliedAt": "timestamp",
  "optimizationMetadata": {
    "model": "string",
    "suggestions": []
  }
}

Execution Analysis

Track failure analyses for learning and improvement.

{
  "analysisId": "uuid",
  "executionId": "uuid",
  "useCaseId": "uuid",
  "analyzedAt": "timestamp",
  "rootCause": "string",
  "suggestions": [],
  "appliedSuggestions": ["suggestionId"],
  "resolutionStatus": "pending" | "resolved" | "dismissed"
}

---

Acceptance Criteria

Feature 1: Instruction Optimizer

• User can click "Optimize" button on any step instruction field
• System sends instruction to Bedrock and receives optimized version
• User sees side-by-side comparison of original vs optimized
• User can apply, cancel, or edit the optimized instruction
• System tracks optimization history
• Rate limiting prevents abuse
• Error states are handled gracefully

Feature 2: Failed Execution Analyzer

• "Analyze Failure" button appears on failed executions
• System analyzes execution log and provides root cause
• System provides categorized suggestions (reorder, modify, add, remove)
• Each suggestion includes reasoning and confidence level
• User can preview changes before applying
• User can apply suggestions individually or in bulk
• System tracks which suggestions were applied
• Analysis results are persisted for future reference

---

Future Enhancements

• A/B testing of optimized vs original instructions
• Learning from successful optimizations to improve prompts
• Batch optimization of multiple steps
• Integration with test execution metrics to measure improvement
• Suggestion templates based on common failure patterns
• Multi-language support for instructions