Generating embeddings

[seoeaa]

how to find out if a file is uploaded to the database so that Generating embeddings does not run constantly

import os
import logging
import gradio as gr
from autollm import AutoQueryEngine
from autollm.utils.document_reading import read_github_repo_as_documents, read_files_as_documents
import lancedb

from pydantic import BaseModel, Field

logging.basicConfig(level=logging.DEBUG)

os.environ["OPENROUTER_API_KEY"] = "sk-or-v1-0c86a53ee96"
model = "openrouter/openai/gpt-3.5-turbo"

relative_folder_path = "examples/data"
#uri = "tmp/lancedb"
#db = lancedb.connect(uri=uri)
#db.drop_database()

documents = read_files_as_documents(input_dir=relative_folder_path , recursive=True)

service_context_params = {
"chunk_size": 512,
"embed_model":'local:intfloat/multilingual-e5-large'
}

llm_params = {
"model": model,
}

query_engine = AutoQueryEngine.from_parameters(
documents=documents,
system_prompt='........................',
query_wrapper_prompt='....................',
enable_cost_calculator=True,
llm_params=llm_params,
service_context_params=service_context_params,
vector_store_params={
"vector_store_type": "LanceDBVectorStore",
"uri": "/tmp/lancedb2",
"table_name": "lancedb",
"nprobs": 20
},
query_engine_params={"similarity_top_k": 10},
)

query = "..................."
response = query_engine.query(query)

print(response.response)

def greet(query):
return query_engine.query(query).response

demo = gr.Interface(fn=greet, inputs="text", outputs="text")
demo.launch()

[fcakyon]

@seoeaa don't provide documents parameter to AutoQueryEngine.from_parameters(, then it will use the previously generated DB instead of generating the embeddings from scratch 👍

[seoeaa]

Thank you

[seoeaa]

4 hours uploaded documents, mbedding and it turned out that it doesn't work Token Usage: 4500
Generating embeddings: 100%|████████████████████████████████████ ████████ ██████████████████████▉| 111480/111487 [4:13:47<00:00, 7.36it/s]Embedding Token Usage: 3161
Generating embeddings: 100%|████████████████████████████████████ ████████ ███████████████████████| 111487/111487 [4:13:48<00:00, 7.32it/s]

[fcakyon]

What is your chunk size, and what is your document type and size?

It seems like you are trying to process 4 pdfs with hundreds of pages.

[seoeaa]

45 files of 100 pages each I wanted to see what would happen

[fcakyon]

It is expected to take that long for 4500 pages of content. You can create separate tables for each file to split the embedding creation step into 4.

[seoeaa]

Maybe each document needs its own table, and the script takes a very long time to start

[fcakyon]

It is expected to take that long for 4500 pages of content. You can create separate tables for each file to split the embedding creation step into 4.

We will improve logging to reflect better to user what autollm is doing at each processing step.