Skip to content

openai_integration_issues.md

Latest commit

 

History

History
201 lines (154 loc) · 5.53 KB

openai_integration_issues.md

File metadata and controls

201 lines (154 loc) · 5.53 KB
Raw

OpenAI Integration Issues - Complete Analysis

Critical Issues Found

1. ❌ Temperature and max_tokens NOT Being Passed to OpenAI API

Location: ai_chatbot_backend/app/dependencies/openai_model.py:120-121

Problem:

request_params = {
    "model": self.model,
    "messages": messages,
    # "temperature": temperature,     # COMMENTED OUT!
    # "max_tokens": max_tokens,        # COMMENTED OUT!
    "stream": stream,
}

Impact:

  • OpenAI uses default temperature=1.0 (more random than intended)
  • max_tokens not set, OpenAI may use a low default → SHORT OUTPUTS
  • Your code sets SAMPLING_PARAMS with max_tokens: 6000 in rag_generation.py:19-24, but they're never passed!

Fix Required: Uncomment lines 120-121 in openai_model.py

2. ❌ LLM Mode Set to "local" Instead of "openai"

Location: ai_chatbot_backend/.env:34-35

Problem:

llm_mode=local          # ← ACTIVE (using vLLM)
#llm_mode = openai      # ← COMMENTED OUT (OpenAI NOT used)

Impact:

  • Server initializes vLLM client, not OpenAI client
  • All requests go to http://128.32.43.210:8001/v1 (your local vLLM server)
  • OpenAI API is NEVER called (confirmed by checking API usage)

Fix Required:

#llm_mode=local
llm_mode=openai

Then restart the server.

3. ⚠️ Potential Model Name Issue

Location: ai_chatbot_backend/.env:48

Current Setting:

OPENAI_MODEL=gpt-5.2

Note: User confirmed this is correct, but if you encounter issues, valid alternatives:

  • gpt-4o (recommended, latest)
  • gpt-4o-mini (cheaper, faster)
  • gpt-4-turbo
  • o1-preview, o1-mini

4. ⚠️ OpenAI Client Not Receiving Sampling Parameters

Location: ai_chatbot_backend/app/services/rag_generation.py:177-183

Problem: When using OpenAIModelClient, the code doesn't pass sampling parameters:

response = engine(
    messages[-1].content,
    messages=remote_messages,
    stream=stream,
    course=course,
    response_format=response_format,
    # Missing: temperature, max_tokens!
)

Impact: Even if we uncomment them in openai_model.py, they won't be passed from rag_generation.py.

Fix Required: Add temperature and max_tokens parameters:

response = engine(
    messages[-1].content,
    messages=remote_messages,
    stream=stream,
    course=course,
    response_format=response_format,
    temperature=SAMPLING_PARAMS["temperature"],  # Add this
    max_tokens=SAMPLING_PARAMS["max_tokens"],     # Add this
)

5. ⚠️ Structured JSON Output May Constrain Length

Location: ai_chatbot_backend/app/services/rag_generation.py:166-169

Current Behavior: When tutor_mode=True, system uses structured JSON schema (RESPONSE_BLOCKS_OPENAI_FORMAT) which enforces strict JSON structure.

Potential Impact: Structured output mode may limit response length or creativity.

Testing: Try with tutor_mode=False to see if responses are longer without JSON constraints.

Summary of Root Causes

Why OpenAI API Is Not Being Called:

  1. ✅ Confirmed: .env has llm_mode=local instead of llm_mode=openai
  2. ✅ Confirmed: User checked OpenAI API usage, no calls are being made

Why Outputs Are Short (Once OpenAI Is Enabled):

  1. ❌ Critical: max_tokens parameter is commented out and not passed to OpenAI
  2. ❌ Critical: temperature parameter is commented out
  3. ⚠️ Possible: Structured JSON schema may constrain output length

Fix Priority

HIGH PRIORITY (Must Fix):

  1. Update .env: Change llm_mode=local to llm_mode=openai
  2. Restart server after .env change
  3. Uncomment parameters in openai_model.py:120-121
  4. Pass parameters in rag_generation.py:177-183

MEDIUM PRIORITY (Should Check):

  1. Verify OPENAI_MODEL=gpt-5.2 is correct (user confirmed)
  2. Test with tutor_mode=False to rule out JSON schema constraints

Verification Steps

After making fixes:

  1. Check server startup logs:

    🤖 LLM Mode: openai
🌐 Setting up OpenAI model pipeline...
✅ OpenAI model pipeline setup successfully! (model: gpt-5.2)

  2. Check request logs during chat:

    [OpenAI Stream] Starting streaming response (chat.completions)...

  3. Check OpenAI API usage dashboard: Should see API calls

  4. Check response length: Should see longer, more complete responses

Code Changes Required

File 1: ai_chatbot_backend/.env

# Change line 34:
#llm_mode=local
llm_mode=openai

File 2: ai_chatbot_backend/app/dependencies/openai_model.py

# Uncomment lines 120-121:
request_params = {
    "model": self.model,
    "messages": messages,
    "temperature": temperature,      # UNCOMMENT THIS
    "max_tokens": max_tokens,         # UNCOMMENT THIS
    "stream": stream,
}

File 3: ai_chatbot_backend/app/services/rag_generation.py

# Add parameters to engine call (line 177-183):
response = engine(
    messages[-1].content,
    messages=remote_messages,
    stream=stream,
    course=course,
    response_format=response_format,
    temperature=SAMPLING_PARAMS["temperature"],  # ADD THIS
    max_tokens=SAMPLING_PARAMS["max_tokens"],     # ADD THIS
)

Additional Notes

  • The system uses a 4-mode architecture (Chat Tutor, Chat Regular, Voice Tutor, Voice Regular)
  • Structured JSON output is used in tutor modes for citation tracking
  • OpenAI API key is present in .env and appears valid
  • vLLM servers are configured at http://128.32.43.210:8001/v1 (currently being used)