Skip to content

DESCRIPTION_FIXES_SUMMARY.md

Latest commit

 

History

History
140 lines (109 loc) · 4.55 KB

DESCRIPTION_FIXES_SUMMARY.md

File metadata and controls

140 lines (109 loc) · 4.55 KB
Raw

Description Extraction Fixes - Summary

Changes Implemented

1. Steam Description Extraction (steam.ts) ✅

Problem: Steam HTML pages truncate content behind "Read more" buttons

  • Only ~700-900 chars available in HTML
  • Full description loaded dynamically via JavaScript

Solution: Use Steam Store API instead of HTML scraping

// Before: HTML scraping (truncated)
const response = await axios.get(`https://store.steampowered.com/app/${appId}`);
const $ = cheerio.load(response.data);
const description = $('#game_area_description').text();
// Result: ~700-900 chars

// After: Steam Store API (full content)
const response = await axios.get('https://store.steampowered.com/api/appdetails', {
  params: { appids: appId }
});
const description = response.data[appId].data?.detailed_description;
// Result: 1,800-12,000+ chars

Results:

Game HTML Scrape Steam API Improvement
Star Wars Jedi: Survivor 739 chars 1,804 chars +144%
Spider-Man Miles Morales 2,472 chars 3,301 chars +34%
God of War Ragnarök 4,809 chars 12,660 chars +163%

2. FitGirl Description Extraction (FitGirlAgent.ts) ✅

Problem: Extraction was getting repack metadata instead of game description

  • Was extracting download links, file sizes, repack features
  • Game description was buried in a specific <div> element

Solution: Look for "Game Description" div specifically

// Before: Generic content extraction
$content.find('p, div').each((_, element) => {
  // Stop at "Repack Features", "Download Mirrors", etc.
  // This missed the game description div
});

// After: Targeted extraction
$content.find('div').each((_, element) => {
  const text = $(element).text().trim();
  if (text.toLowerCase().startsWith('game description')) {
    // This is the actual game description!
    gameDescription = text;
  }
});

Results:

Game Old Extraction New Extraction Similarity with Steam
Star Wars Jedi: Survivor 5,000 chars (metadata) 716 chars (description) 72.4%
God of War Ragnarök 5,000 chars (metadata) 5,063 chars (description) 91.5%

Final Test Results

Games Now Matching with 100/100 Score:

Game Previous Score New Score Status
Marvel's Spider-Man: Miles Morales 45/100 100/100 ✅ Fixed
Star Wars Jedi: Survivor 78/100 100/100 ✅ Improved
God of War (2018) 100/100 100/100 ✅ Still working
God of War Ragnarök 85/100 85/100 ✅ Still working

Match Reason Examples:

Spider-Man Miles Morales (100/100):

- title contains game name
- matches alternative title
- very high word match ratio
- all main keywords present
- strong Steam description match  <-- NEW!

Star Wars Jedi: Survivor (100/100):

- exact phrase in title
- title contains game name
- strong title match
- matches alternative title
- very high word match ratio
- all main keywords present
- strong Steam description match  <-- NEW!

Files Modified

  1. apps/api/src/utils/steam.ts

    • Replaced HTML scraping with Steam Store API
    • Added proper TypeScript interfaces
    • Improved error handling

  2. apps/api/src/services/games/search-agents/FitGirlAgent.ts

    • Added "Game Description" div detection
    • Prioritized game description over repack metadata
    • Maintained fallback to previous method

Testing Evidence

All test data saved in description-test-results/:

  • *_steam_full.html - Raw Steam HTML pages
  • *_steam_api_*.txt - Steam API responses
  • *_fitgirl_full.html - Raw FitGirl HTML pages
  • *_fitgirl_game_desc.txt - Extracted game descriptions
  • _summary.json - Comparison metrics

Impact on Description Similarity Scoring

The improved extraction directly affects the description similarity scoring:

Before:

  • Steam: Truncated descriptions (~700 chars)
  • FitGirl: Repack metadata (~5000 chars of irrelevant content)
  • Similarity: 0.9% - 6.3% (very low)

After:

  • Steam: Full descriptions via API (1800-12000 chars)
  • FitGirl: Actual game descriptions (700-5000 chars)
  • Similarity: 72.4% - 91.5% (high)

This results in the "strong Steam description match" bonus (+40 points) being correctly applied.

Notes

  • Steam Store API has rate limiting (appears to be ~200 requests/5 minutes)
  • FitGirl pages vary in structure; the "Game Description" div is the most reliable marker
  • Some games (Hogwarts Legacy, Spider-Man Remastered) are not on FitGirl