Skip to content

fix: resolve Python FFI buffer limits and single-quote escaping in tool calling #760

Description

@ApurveKaranwal

Description

While working with the Cactus codebase, I encountered three related bugs/limitations in the FFI bindings and tool call parsing logic:

1. Invalid JSON generated during single-quoted parameter parsing in tool calls

LiquidAI models (LFM2/LFM2.5) format tool calls as Python-like arguments (e.g., my_tool(param='val')).

  • The issue: In cactus-engine/src/chat_tools.h, the parser function py_literal_to_json translates escaped single quotes (\') literally as \' inside the resulting double-quoted JSON string (producing "hello \'world\"").
  • Why it fails: Per RFC 8259, \' is an invalid escape sequence in JSON. Standard JSON parsers (e.g. C++ picojson and python's json module) fail with a syntax error, causing tool execution to abort.
  • Proposed fix: Modify the escape sequence parser to unescape \' to ' in the double-quoted JSON string since single quotes do not require escaping.

2. RuntimeError when retrieving documents > 4KB from index

  • The issue: The vector database supports indexing documents up to 65,535 bytes (Index::validate_documents). However, the Python FFI binding cactus_index_get uses a hardcoded buffer limit _INDEX_DOC_BUF_SIZE = 4096.
  • Why it fails: When retrieving a document that exceeds 4KB, the C++ code returns -1 (buffer too small), causing the Python wrapper to raise RuntimeError: Failed to get from index.
  • Proposed fix: Bump _INDEX_DOC_BUF_SIZE to 65536 in the Python FFI bindings to accommodate the maximum supported document size.

3. RAG query response buffer size limitation (64KB)

  • The issue: The Python binding cactus_rag_query allocates a 65536 byte buffer.
  • Why it fails: RAG queries returning multiple chunks (up to top_k = 5) can easily exceed 64KB, causing query failures.
  • Proposed fix: Increase the RAG query buffer to 1 << 20 (1MB), matching the completion buffer size.

Verification

  • Created a new C++ test suite cactus-engine/tests/test_chat_tools.cpp to verify parsing logic.
  • Verified that all tests compile and pass successfully.

I have these changes ready on a local branch. Please let me know if you would like me to submit a Pull Request!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions