Problem of not exactly embedding csv file with FAISS

[jsstar522]

Hello

I'm trying to embed a CSV file and implement a search function using FAISS. The CSV file loads correctly, but there's an issue where only part of the CSV is being embedded.

csv file

name,population
a,2288
b,5280
c,8051
d,7414
e,2752
f,4960
g,2290
h,5782
i,229

I'm embedding documents and asking questions as shown below, but it seems like the CSV file is being embedded incorrectly.

code

lm = Ollama(model='llama3')

DB_FAISS_PATH = 'vectorstore/db_faiss'

loader = CSVLoader(file_path="tmp/output.csv", encoding="utf-8", csv_args={'delimiter': ','})
data = loader.load()

print("*************************************")
print(data)
print("*************************************")

embeddings = HuggingFaceEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2', model_kwargs={'device': 'cpu'})
db = FAISS.from_documents(data, embeddings)
db.save_local(DB_FAISS_PATH)
chain = ConversationalRetrievalChain.from_llm(llm=llm, retriever=db.as_retriever())
print(chain({"question": "let me show all of data and how many is data", "chat_history": []}))

Question

let me show all of data and how many is data

Return

*************************************
[Document(metadata={'source': 'tmp/output.csv', 'row': 0}, page_content='name: a\npopulation: 2288'), Document(metadata={'source': 'tmp/output.csv', 'row': 1}, page_content='name: b\npopulation: 5280'), Document(metadata={'source': 'tmp/output.csv', 'row': 2}, page_content='name: c\npopulation: 8051'), Document(metadata={'source': 'tmp/output.csv', 'row': 3}, page_content='name: d\npopulation: 7414'), Document(metadata={'source': 'tmp/output.csv', 'row': 4}, page_content='name: e\npopulation: 2752'), Document(metadata={'source': 'tmp/output.csv', 'row': 5}, page_content='name: f\npopulation: 4960'), Document(metadata={'source': 'tmp/output.csv', 'row': 6}, page_content='name: g\npopulation: 2290'), Document(metadata={'source': 'tmp/output.csv', 'row': 7}, page_content='name: h\npopulation: 5782'), Document(metadata={'source': 'tmp/output.csv', 'row': 8}, page_content='name: i\npopulation: 229')]
*************************************
{'question': 'let me show all of data and how many is data', 'chat_history': [], 'answer': 'Let me help you!\n\nThe given context shows the population for each name:\n\n1. d - 7414\n2. i - 229\n3. a - 2288\n4. e - 2752\n\nThere are 4 pieces of data in total.'}

Based on the results, the CSV Loader is loading the file correctly. However, when I ask the chain for the entire data, it only returns 4 records. Am I missing something?

[mdouze]

Related to a wrapping library above Faiss.