v0.2.5: Fix organize silently overwriting files with duplicate names

execute_plan() computed its destination as `cat_dir / new_name` with no
existence check and copied with shutil.copy2(), which overwrites. When the
AI plan assigned the same category + new_name to two different files, only
one survived on disk -- while stats["copied"] counted both and _summary.md
listed both as organized successfully. Silent data loss with a success
report.

Dedup masks this in practice: compare --link converts identical content to
symlinks before organize runs, so byte-identical files rarely collide. The
exposure is genuinely different content earning the same name -- two
unrelated shopping lists both becoming shopping-list.txt. The defect has
been present since organize was introduced.

Found while designing batched AI backends, where it becomes much more
likely: a backend processing files in batches has no view of the names
earlier batches already assigned.

Collision handling:
- _resolve_unique_dest() claims a destination no other entry in the same
  run is using, appending -2, -3, ... before the extension so file type is
  preserved (script.py -> script-2.py)
- Keys on the lowercased path, so Notes.txt and notes.txt are treated as a
  collision on Windows rather than one silently winning
- Tracking is scoped to a single run, never checked against files already
  on disk. Checking the filesystem would break idempotency: re-running
  organize would produce -2, -3 copies on every pass instead of
  overwriting its own previous output
- Every rename is reported in three places -- the summary counts, a new
  "Renamed to avoid collisions" section in _summary.md, and the per-file
  mapping. A silent rename is nearly as bad as a silent overwrite, because
  the summary would stop describing the disk
- stats["renamed"] added and surfaced in the CLI output

Policy is auto-rename and report, not abort. One name clash must not
discard an otherwise complete run; the invariant that matters is that no
file is lost, not that no name repeats.

Plan validation:
- validate_plan(plan, base_dir, expected_sources=None) checks a plan
  before anything touches the filesystem: every source must exist, no
  source may appear twice, every expected source must be present, and
  duplicate destinations are reported
- organize now refuses to run when a plan would lose or mis-file content.
  Duplicate destinations are informational (execute_plan handles them);
  missing, duplicated, and nonexistent sources are fatal

Prompt:
- prompts/organize.md had no uniqueness requirement at all. "Every file
  MUST get its own new_name" meant "not null", as the following clause
  says. Added an explicit rule that every category + new_name pair must be
  unique, and that similar files be distinguished by content rather than
  by numeric suffix

New tests (50 total, up from 36) -- first automated suite in tests/, which
previously held only one-offs/ proof-of-concept scripts:
- test_colliding_names_do_not_lose_content: reproduces the defect; two
  entries sharing a destination must yield two files with both contents
- test_collision_rename_is_reported: renames appear in the summary
- test_collision_across_categories_is_not_a_collision: the same filename
  in different categories is legitimate and must not be renamed
- test_extension_preserved_when_de_colliding: script.py -> script-2.py
- test_case_only_difference_collides_on_windows: Notes.txt vs notes.txt
- test_rerun_does_not_accumulate_duplicates: idempotency guard
- test_many_collisions_all_survive: a 12-way collision keeps all 12 files
- test_rename_counter_is_reported_in_stats
- test_validate_plan_*: duplicate, missing, and nonexistent sources
- no-regression guards on the ordinary non-colliding path

Also adds tests/checklists/ with a human test checklist covering what
mocks cannot: whether _summary.md reads clearly, real Claude CLI output,
Windows case-insensitivity, whether the refusal message is actionable, and
re-run idempotency against a real organized folder.

Design: 2026-07-25__14-07-53__editor-mode-architecture-and-sublime-archaeology.md
Design: 2026-07-26__03-15-17__addendum__local-model-backend-and-collision-defect.md

Author: djdarcy

CHANGELOG.md

@@ -2,6 +2,52 @@
 
 All notable changes to notepad-cleanup will be documented in this file.
 
+## [0.2.5] - 2026-07-26
+
+### Fixed
+- **`organize` no longer silently overwrites files when two get the same name.**
+  `execute_plan()` computed the destination path with no existence check and used
+  `shutil.copy2()`, which overwrites. When the AI plan assigned the same
+  `category` + `new_name` to two different files, only one survived on disk --
+  while `stats["copied"]` counted both and `_summary.md` reported both as
+  organized successfully. Silent data loss with a success report
+- Colliding destinations are now de-collided (`name.txt` -> `name-2.txt`), keeping
+  every file, and each rename is reported in three places: the summary counts, a
+  dedicated "Renamed to avoid collisions" section, and the per-file mapping.
+  Policy is auto-rename and report -- never silent, and never abort a whole run
+  over a name clash, because the invariant that matters is *no file is lost*
+- Collision tracking is scoped to a single run, so re-running `organize` still
+  overwrites its own previous output rather than accumulating `-2`, `-3` copies
+
+### Added
+- `validate_plan(plan, base_dir, expected_sources=None)` -- checks a plan before
+  any file is touched: sources must exist, no source may appear twice, every
+  expected source must be present, and duplicate destinations are flagged.
+  `organize` now refuses to run when a plan would lose or mis-file content
+- `stats["renamed"]` counts collision renames, surfaced in the CLI output
+- Uniqueness rules in `prompts/organize.md`: every `category` + `new_name` pair
+  must be unique, and similar files must be distinguished by content rather than
+  by numeric suffix. The prompt previously had no uniqueness requirement at all
+
+### Tests
+- `tests/test_organize_collisions.py` -- first automated regression suite in
+  `tests/` (previously only `tests/one-offs/` POC scripts existed):
+  - `test_colliding_names_do_not_lose_content`: reproduces the defect; two
+    entries sharing a destination must yield two files with both contents
+  - `test_collision_rename_is_reported`: renames must appear in the summary
+  - `test_collision_across_categories_is_not_a_collision`: same filename in
+    different categories is legitimate and must not be renamed
+  - `test_extension_preserved_when_de_colliding`: `script.py` -> `script-2.py`
+  - `test_many_collisions_all_survive`: 12-way collision keeps all 12 files
+  - `test_case_only_difference_collides_on_windows`: `Notes.txt` vs `notes.txt`
+    collide on a case-insensitive filesystem; both must survive
+  - `test_rerun_does_not_accumulate_duplicates`: idempotency guard -- a second
+    `organize` run must overwrite its own output, not create `-2` copies
+  - `test_rename_counter_is_reported_in_stats`: `stats['renamed']` is accurate
+  - `test_validate_plan_*`: duplicate, missing, and nonexistent sources
+  - no-regression guards on the ordinary non-colliding path
+  - suite total: 14 new tests; **50 passed** overall (was 36)
+
 ## [0.2.4] - 2026-04-10
 
 ### Fixed

notepad_cleanup/_version.py

@@ -21,13 +21,13 @@
 # Version components - edit these for version bumps
 MAJOR = 0
 MINOR = 2
-PATCH = 4
+PATCH = 5
 PHASE = None  # Per-MINOR feature set: None, "alpha", "beta", "rc1", etc.
 PRE_RELEASE_NUM = 0  # PEP 440 pre-release number (e.g., a1, b2)
 PROJECT_PHASE = "prealpha"  # Project-wide: "prealpha", "alpha", "beta", "stable"
 
 # Auto-updated by git hooks - do not edit manually
-__version__ = "0.2.4_main_16-20260410-34ed8c9d"
+__version__ = "0.2.5_main_16-20260726-450e79e2"
 __app_name__ = "notepad-cleanup"
 
 

notepad_cleanup/cli.py

@@ -13,7 +13,7 @@
 from .extractor import extract_phase1, extract_phase2, merge_results
 from .saver import save_extraction
 from .organizer import (generate_prompt, invoke_claude_cli, save_prompt_to_file,
-                         find_claude_cli, parse_plan, execute_plan,
+                         find_claude_cli, parse_plan, execute_plan, validate_plan,
                          separate_links, join_links)
 from .dedup import (find_session_dirs, build_hash_index, find_duplicates,
                     find_content_files,
@@ -614,6 +614,23 @@ def organize(folder, last, backend, dry_run, verbose):
 
     console.print(f"  Claude proposed a plan for [bold]{len(plan)}[/bold] files\n")
 
+    # Step 2b: Validate the plan before touching the filesystem. Duplicate or
+    # missing sources mean content would be mis-filed or left unorganized --
+    # which matters most because later steps may delete the origin.
+    problems = validate_plan(plan, folder)
+    if problems:
+        fatal = [p for p in problems if "auto-renamed" not in p]
+        for p in problems:
+            style = "yellow" if "auto-renamed" in p else "red"
+            console.print(f"  [{style}]plan issue:[/{style}] {p}")
+        console.print("")
+        if fatal:
+            console.print(
+                "[red]Refusing to organize: the plan has problems that would "
+                "lose or mis-file content.[/red]")
+            console.print(f"  Raw output saved to: {log_file}\n")
+            return
+
     # Step 3: Execute the plan locally (link-aware)
     with console.status("Organizing files..."):
         summary, stats = execute_plan(plan, folder, linked_paths=linked_paths)
@@ -622,6 +639,9 @@ def organize(folder, last, backend, dry_run, verbose):
     console.print(f"  Files copied:     {stats['copied']}")
     if stats.get('linked', 0):
         console.print(f"  Files linked:     {stats['linked']} (symlink/hardlink to canonical)")
+    if stats.get('renamed', 0):
+        console.print(f"  [yellow]Renamed:          {stats['renamed']} "
+                      f"(name collisions -- see _summary.md)[/yellow]")
     if stats['errors']:
         console.print(f"  [red]Errors:         {stats['errors']}[/red]")
 

notepad_cleanup/organizer.py

@@ -304,6 +304,108 @@ def _write_organized_link_manifest(entries, organized_dir):
     )
 
 
+def _resolve_unique_dest(cat_dir, new_name, used):
+    """Pick a destination path that no other entry in THIS run has claimed.
+
+    Two plan entries can legitimately propose the same `new_name` -- an AI
+    backend batching files has no global view, and even a single-shot backend
+    may name two different grocery lists "grocery-list.txt". Copying both to
+    the same path silently destroys the first one, so we de-collide instead.
+
+    Scope note: collisions are tracked only WITHIN a run (via `used`), never
+    against files already on disk. Checking the filesystem would break
+    idempotency -- re-running organize would produce name-2, name-3, ... on
+    every pass instead of overwriting its own previous output.
+
+    Returns: (dest_path, final_name, original_name_if_renamed)
+    """
+    stem = Path(new_name).stem
+    suffix = Path(new_name).suffix
+
+    candidate = new_name
+    n = 1
+    while True:
+        dest = cat_dir / candidate
+        key = str(dest).lower()          # Windows paths are case-insensitive
+        if key not in used:
+            used.add(key)
+            return dest, candidate, (new_name if candidate != new_name else None)
+        n += 1
+        candidate = f"{stem}-{n}{suffix}"
+
+
+def validate_plan(plan, base_dir, expected_sources=None):
+    """Check a plan for defects that would lose or mis-file content.
+
+    Run this BEFORE execute_plan and refuse to proceed on any problem. A file
+    missing from the plan is a file left unorganized, and a plan referencing a
+    source that doesn't exist means the AI backend invented a filename -- both
+    are worth catching before anything touches the filesystem.
+
+    Checks:
+      - every `source` exists on disk
+      - no `source` appears more than once
+      - every expected source is present (when `expected_sources` is given)
+      - duplicate category/new_name pairs are reported (informational -- these
+        are auto-renamed by execute_plan, not fatal)
+
+    Args:
+        plan: list of plan entry dicts
+        base_dir: extraction directory the sources are relative to
+        expected_sources: optional iterable of relpaths that MUST appear
+
+    Returns: list of human-readable problem strings ([] when the plan is clean)
+    """
+    base_dir = Path(base_dir)
+    problems = []
+
+    seen_sources = {}
+    for i, entry in enumerate(plan):
+        source = (entry.get("source") or "").replace("\\", "/")
+        if not source:
+            problems.append(f"entry {i}: missing 'source'")
+            continue
+
+        seen_sources.setdefault(source, []).append(i)
+
+        if not (base_dir / source).exists():
+            problems.append(
+                f"entry {i}: source does not exist on disk: {source}"
+            )
+
+    for source, idxs in sorted(seen_sources.items()):
+        if len(idxs) > 1:
+            problems.append(
+                f"duplicate source: {source} appears {len(idxs)} times "
+                f"(entries {idxs})"
+            )
+
+    if expected_sources is not None:
+        expected = {str(s).replace("\\", "/") for s in expected_sources}
+        missing = sorted(expected - set(seen_sources))
+        for m in missing:
+            problems.append(f"missing from plan: {m}")
+
+    # Informational: execute_plan de-collides these, but a backend producing
+    # many of them signals the prompt isn't being told what names exist yet.
+    dests = {}
+    for i, entry in enumerate(plan):
+        key = (
+            (entry.get("category") or "misc"),
+            (entry.get("new_name") or ""),
+        )
+        if key[1]:
+            dests.setdefault(key, []).append(i)
+    for (cat, name), idxs in sorted(dests.items()):
+        if len(idxs) > 1:
+            problems.append(
+                f"duplicate destination (will be auto-renamed): "
+                f"{cat}/{name} claimed by entries {idxs}"
+            )
+
+    return problems
+
+
 def execute_plan(plan, base_dir, linked_paths=None):
     """
     Execute the organization plan by copying/renaming files.
@@ -324,7 +426,7 @@ def execute_plan(plan, base_dir, linked_paths=None):
     organized_dir = base_dir / "organized"
     organized_dir.mkdir(exist_ok=True)
 
-    stats = {"copied": 0, "linked": 0, "errors": 0}
+    stats = {"copied": 0, "linked": 0, "errors": 0, "renamed": 0}
     categories = {}
     details = []
     organized_links = []  # Track which files were linked vs copied
@@ -333,6 +435,11 @@ def execute_plan(plan, base_dir, linked_paths=None):
     from .dedup import _can_create_symlink
     symlink_ok = _can_create_symlink()
 
+    # Destinations claimed during THIS run -- prevents one entry from silently
+    # overwriting another when two entries propose the same category/new_name.
+    used_dests = set()
+    renames = []
+
     for entry in plan:
         source = entry.get("source", "")
         category = entry.get("category", "misc")
@@ -353,7 +460,16 @@ def execute_plan(plan, base_dir, linked_paths=None):
         cat_dir = organized_dir / category
         cat_dir.mkdir(exist_ok=True)
 
-        dest_path = cat_dir / new_name
+        # Claim a destination no other entry in this run is using. Renaming is
+        # always reported -- a silent rename is nearly as bad as a silent
+        # overwrite, because the summary would no longer describe the disk.
+        dest_path, new_name, renamed_from = _resolve_unique_dest(
+            cat_dir, new_name, used_dests)
+        rename_note = ""
+        if renamed_from:
+            renames.append((source, category, renamed_from, new_name))
+            stats["renamed"] += 1
+            rename_note = f" [renamed from {renamed_from} -- name collision]"
 
         # Check if this file is a dedup link
         source_resolved = source_path.resolve()
@@ -366,7 +482,8 @@ def execute_plan(plan, base_dir, linked_paths=None):
             if ok:
                 stats["linked"] += 1
                 categories.setdefault(category, []).append(new_name)
-                details.append(f"  {source} -> {category}/{new_name} [{link_type}]")
+                details.append(
+                    f"  {source} -> {category}/{new_name} [{link_type}]{rename_note}")
                 organized_links.append({
                     "rel_path": f"{category}/{new_name}",
                     "canonical": str(canonical),
@@ -378,7 +495,8 @@ def execute_plan(plan, base_dir, linked_paths=None):
                     shutil.copy2(str(source_path), str(dest_path))
                     stats["copied"] += 1
                     categories.setdefault(category, []).append(new_name)
-                    details.append(f"  {source} -> {category}/{new_name} [copy fallback]")
+                    details.append(
+                        f"  {source} -> {category}/{new_name} [copy fallback]{rename_note}")
                 except Exception as e:
                     stats["errors"] += 1
                     details.append(f"  ERROR copying {source}: {e}")
@@ -388,7 +506,7 @@ def execute_plan(plan, base_dir, linked_paths=None):
                 shutil.copy2(str(source_path), str(dest_path))
                 stats["copied"] += 1
                 categories.setdefault(category, []).append(new_name)
-                details.append(f"  {source} -> {category}/{new_name}")
+                details.append(f"  {source} -> {category}/{new_name}{rename_note}")
             except Exception as e:
                 stats["errors"] += 1
                 details.append(f"  ERROR copying {source}: {e}")
@@ -405,8 +523,19 @@ def execute_plan(plan, base_dir, linked_paths=None):
     summary_lines.append(f"- **Categories**: {len(categories)}")
     if stats["errors"]:
         summary_lines.append(f"- **Errors**: {stats['errors']}")
+    if renames:
+        summary_lines.append(f"- **Renamed (name collisions)**: {len(renames)}")
     summary_lines.append("")
 
+    if renames:
+        summary_lines.append("## Renamed to avoid collisions\n")
+        summary_lines.append(
+            "Two or more files were assigned the same name. Each was kept as a "
+            "separate file rather than overwritten:\n")
+        for source, cat, was, now in renames:
+            summary_lines.append(f"- `{source}`: `{cat}/{was}` -> `{cat}/{now}`")
+        summary_lines.append("")
+
     summary_lines.append("## Categories\n")
     for cat, files in sorted(categories.items()):
         summary_lines.append(f"### {cat} ({len(files)} files)")

notepad_cleanup/prompts/organize.md

@@ -39,6 +39,8 @@ Return ONLY a JSON array. No markdown fencing, no explanation, just raw JSON:
 ## Rules for naming and categorizing
 
 - Every file MUST get its own `new_name` — never set it to null
+- **Every `category` + `new_name` pair MUST be unique across the whole plan.** Two files may share a category, and the same filename may appear in *different* categories, but no two entries may target the same `category/new_name`. If two files are genuinely similar, distinguish them by content (`grocery-list-june.txt` and `grocery-list-hardware-store.txt`), not by a numeric suffix
+- If you cannot tell two files apart well enough to name them distinctly, still give them different names — a duplicated name means one file would overwrite the other
 - Each tab is preserved as an individual file, even short notes
 - Use lowercase-with-dashes for folder and file names
 - Keep `.txt` for plain text and general notes