The above image was made with stable diffusion using the prompt 'letters falling from the sky.'
I'm mildly obsessed with news article recommendation algorithms. At WantToKnow.info, we have an archive of 12k news article summaries. Three-fourths of these are on high-level corruption and cover-ups, while the remainder are on inspiring topics. For fun on my own time, I've been playing around with a copy of our news archive, exploring relationships between articles based on article text.
My first post on the project described my initial encounter with the data. My second post described the trials and tribulations of preprocessing. My third post showed how I finally got all of the data into a pandas dataframe. My fourth post showed how I scrubbed and standardized the data to make deeper analysis possible.
Our current data has several features: article date, upload date, ID number, title, text, publication, category tags, and manually-assigned rating. Right now on our website, article searches can be sorted by article date, upload date, and rating. This is great, but it would be better if sorting by publication were also possible. One happy byproduct of my project so far has been a publication standardization key for cleaning up our messy archive publication data.
These features on their own would be sufficient to generate a list of recommended articles for a selected article. But there's other good information buried in the data. So I decided to use a natural language processing technique that turns article text into mathematical coordinates and then measures the distance between these sets of coordinates. What I ended up with was a list of the ten closest matches for every article.
First, I had some cleanup to do on my preprocessed file. When all was said and done, about 10% of the entries were trimmed from the dataframe, mostly due to missing essential data. Then I used Scikit Learn to turn the article text into TF-IDF (Term Frequency - Inverse Document Frequency) vectors. After that, I computed the cosine similarity for all of the article vectors, and wrote a list of the ten closest articles to a new column in my dataframe.
The next step is to test out the results to see if the articles on the list are actually good recommendations. I'm still thinking about the best way to do this. Realistically, I'll probably just randomly pick a dozen to check manually. Even if the matches aren't great, I've added a nifty new feature to the data.
Below is my code. Note that this was computationally intensive and took several minutes to run.
import pandas as pd
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
#read in selected columns of preprocessed file
df = pd.read_csv("C:\\datasources\\WTKfullraw.csv", sep='|', usecols=['ArticleId','Title','PublicationDate','cpub','Links','wtkURL','Description','Note','tags','Priority'])
#deduplication and NaN cleanup
df.drop_duplicates('Title')
df = df[df['tags'].notna()]
df = df[df['Priority'].notna()]
df = df[df['wtkURL'].notna()]
#substituting multiple spaces with single space
df['Description']= df['Description'].apply(lambda x: re.sub(r'\s+',' ', str(x)))
#remove double quotes
df['Description']= df['Description'].apply(lambda r: r.replace('\"\"', '\"'))
#remove paragraph styling
df['Description']= df['Description'].apply(lambda r: r.replace(''
, ''
))
#compute the TF-IDF vectors for the preprocessed article text
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(df['Description'])
#compute the cosine similarity matrix
cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)
#iterate through each article and find the most similar articles based on cosine similarity
related_articles = []
for i in range(len(df)):
similar_articles = []
# get the cosine similarity scores for the current article
scores = list(enumerate(cosine_sim[i]))
# sort the scores in descending order
scores = sorted(scores, key=lambda x: x[1], reverse=True)
# get the top 10 most similar articles (excluding itself)
top_similar = scores[1:11]
for j, _ in top_similar:
similar_articles.append(df.iloc[j]['ArticleId'])
related_articles.append(similar_articles)
#add the column of related articles to the DataFrame
df['Related'] = related_articles
#df.tail(6)
#df.to_csv('C:\\datasources\\WTKrelated.csv', index=False)
Until this point in the project, I was mostly revising code from five years ago, when I last worked on this archive. But today's code is all new. Now I'm trying to decide if I want to get serious enough about Pyscript to attempt making a webpage from this data. Right now, I'm on the fence.
Read my novels:
See my NFTs: