The above image was made with stable diffusion using the prompt 'closeup blue python and a magnifying glass.'
Last year, I wrote a news recommendation algorithm for WantToKnow.info. You can read about the project here and test out the recommendations by clicking on any article title in our archive. The recommendations are based on something called TF-IDF vector cosine similarity, which is to say the mathematical relationships between news stories.
The recommendations generated by my code are great so I started wondering about expanding the underlying tech to vector search. WantToKnow has good search already, but it's keyword based. My thinking is that vectorizing search queries and then comparing query vectors with news article vectors could potentially surface good stories in situations where keywords alone aren't cutting it.
Yesterday, I decided to get into vector search for real. My goal is to make a web page that takes any detailed question or description about any conspiracy-related topic as input and outputs a list of the 20 most relevant news article summaries. As a fun added constraint, I want to make this search app front end only, and I want to write all of the logic using python in the html with Pyscript. For this project, instead of a database, I plan to use a csv file stored on IPFS.
Diving in, I created a pandas dataframe of our news archive, then computed TF-IDF vectors for all articles, storing these vectors in a new column. Here's the code I started with:
import pandas as pd
import numpy as np
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
df = pd.read_csv("C:\\datasources\\ArticleScrubbed.csv", sep='|', usecols=['ArticleId','Title','PublicationDate','Publication','Links','Description','Priority','url'])
#deduplication and NaN cleanup
df.drop_duplicates('Title')
df = df[df['Priority'].notna()]
#substituting multiple spaces with single space
df['Description']= df['Description'].apply(lambda x: re.sub(r'\s+',' ', str(x)))
#Compute the TF-IDF vectors for the preprocessed article text
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(df['Description'])
df['Vector'] = tfidf_matrix.toarray().tolist()
df.tail(6)
The code executed without incident, but when I wrote the dataframe to a new csv file, it took forever. Then I looked at the size of the file and it was 3.5GB. Without the vectors, the entire news archive is only about 27MB. And suddenly it became clear why everybody didn't use vector search.
Now I'm left wondering if it would be better to proceed with the pre-computed vectors or to use browser resources to calculate and compare vectors on the fly. Neither path is optimal as both will add GB worth of load to user systems. All I really want to do is get to the point where I can test vector search to see if it might be worth it to implement on our site. I'm sure I'll get there, but it might be a bumpy ride.
Read Free Mind Gazette on Substack
Read my novels:
See my NFTs: