The above image was made with stable diffusion using the prompt 'sorting colored boxes into rows.'
I want to start playing around with news article recommendation algorithms. I'm also interested in using machine intelligence to identify and map out connections between news reports to see what larger stories emerge. This is related to my work for WantToKnow.info, but I'm just doing it for fun on my own time.
My first post on the project described my initial encounter with the data. My second post described the trials and tribulations of preprocessing. My third post showed how I finally got all of the data into a pandas dataframe. Here, you'll find the code I used to scrub and standardize the data to make deeper analysis possible.
The code below does many things. It expands category tag lists into new columns, separates article summaries from notes, and extracts links from both summaries and notes. It also simplifies publication data with both one-off anonymous functions and the mediacondense dictionary I build in the precious step. Here it is:
from bs4 import BeautifulSoup as bs
import pandas as pd
import numpy as np
import os
import re
#read in preprocessed file
dfP = pd.read_csv("C:\\datasources\\WTKpreprocessed.csv", sep='|', encoding='utf-8')
#split raw story into summary and note.
dfP['Summary'], dfP['Note'] = dfP['Description'].str.split('Note:', 1).str
#create unique tag columns to make values quickly mappable for graph generation
dfP['tags'] = dfP['tags'].astype(str).apply(lambda v: v.replace('\'', ''))
dfP['tags'] = dfP['tags'].astype(str).apply(lambda v: v.replace('[[', '['))
dfP['tags'] = dfP['tags'].astype(str).apply(lambda v: v.replace(']]', ']'))
dfP.tags = dfP.tags.str[1:-1].str.split(',').tolist()
dfT = pd.DataFrame(dfP.tags.values.tolist(), dfP.index).add_prefix('tag_')
form = lambda f: 'tag_{}'.format(f + 1)
pd.DataFrame(
dfP.tags.values.tolist(),
dfP.index, dtype=object
).fillna('').rename(columns=form)
df = pd.concat([dfP, dfT], axis=1)
#add tagcount column for convenient advanced indexing
df['tagcount'] = df.tags.apply(lambda l: len(l))
#initialize some lists for data extraction and cleaning
Quotes = []
Summaries = []
Note_text = []
Adtl_ref_links = []
Note_ref_links = []
Media_sources = []
#set variables up for parsing markup
stories = df['Summary'].astype(str).values.tolist()
notes = df['Note'].astype(str).values.tolist()
medias = df['Publication'].astype(str).values.tolist()
#some loops to extract desired text and links
for i in stories:
c = bs(i,'lxml')
try:
quote = c.strong.text
except (AttributeError):
quote = c.text.split('. ')[0]
summary = c.text
ref_links = []
for n in c.find_all('a'):
xlinks = n.get('href')
ref_links.append(xlinks)
Quotes.append(quote)
Summaries.append(summary)
Adtl_ref_links.append(ref_links)
#switch parser from lxml for notes parsing because different parsers are good for different things
for i in notes:
c = bs(i,'html.parser')
note = c.text
note_links = []
for n in c.find_all('a'):
nlinks = n.get('href')
note_links.append(nlinks)
Note_text.append(note)
Note_ref_links.append(note_links)
#initial cleanup of pub. bs4 282 User Warning may show up because some media sources are urls. no biggie
for i in medias:
c = bs(i,'html.parser')
media = c.text
Media_sources.append(media)
#make new dataframe from extracted data
df3 = pd.DataFrame({'quote': Quotes,
'description': Summaries,
'note': Note_text,
'pub': Media_sources,
'Summary_ref_links': Adtl_ref_links,
'Note_links': Note_ref_links})
#join new dataframe to big raw dataframe
dfR = pd.concat([df, df3], axis=1)
dfR.drop(['Publication', 'Summary', 'url', 'url2'], axis=1, inplace=True)
#begin cleanup of media sources
dfR['pub'], dfR['pub2'] = dfR['pub'].str.split('/', 1).str
dfR['pub'], dfR['pubdetail'] = dfR['pub'].str.split('(', 1).str
dfR['pubdetail'] = dfR['pubdetail'].astype(str).apply(lambda u: u.strip(')'))
dfR['pub'] = dfR['pub'].str.strip()
dfR['pub'] = dfR['pub'].astype(str).apply(lambda u: u.strip('"'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('\'',''))
#use dfR.pub.unique() with pd.set_option('display.max_colwidth', 1000) to view raw media sources list
#dfR['pub'].value_counts()
#combine variations of media source into single item
#one-off translations are easy, but should eventually be moved to mediacondense list
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Seattle times', 'Seattle Times'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Scientific American Blog', 'Scientific American'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('The Sacramento Bee', 'Sacramento Bee'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Mother Jones Magazine', 'Mother Jones'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Sydney Mountain Herald', 'Sydney Morning Herald'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('The Nation magazine', 'The Nation'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Chicago Sun-Times News Group', 'Chicago Sun-Times'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Wired magazine', 'Wired'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('The New Yorker magazine', 'The New Yorker'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Fortune magazine', 'Fortune'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('The Atlantic Monthly', 'The Atlantic'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('Tikkun Magazine - March', 'Tikkun Magazine'))
dfR['pub'] = dfR['pub'].astype(str).apply(lambda r: r.replace('The Daily Mail', 'Daily Mail'))
#dfR.pub.head(3)
#read in preprocessed mediacondense file
mkeys = pd.read_csv("C:\\datasources\\mediakeys.csv", sep='|', usecols=['asis','clean'], encoding='utf-8')
mkeys.set_index('asis', inplace=True)
dx = pd.Series(mkeys.clean.values,index=mkeys.index).to_dict()
dfR['cpub'] = dfR['pub'].map(dx).fillna(dfR['pub'])
dfR.head(3)
#save full raw dataframe
dfR.to_csv("C:\\datasources\\WTKfullraw.csv", sep='|', index=False, encoding='utf-8')
Now that the data has been scrubbed and standardized, it's time to take a step back and think deeply about the algos I want to create. I have a pretty good idea of what I want to do, but I'm feeling like I should be thinking more about the links contained in our summaries and notes. The note links in particular could in many cases already recommend related stories in our archive, but the use of such links is by no means standardized.
I also need to think more about the end use. When I started this project, I was sort of picturing just translating any algo I come up with into javascript, but something tells me it won't be that easy, especially if I end up doing more advanced natural language processing. So there's a lot to think about. In the meantime, I'll probably get back to my usual blogging subjects soon.
Read my novels:
See my NFTs: