There are several methods of generating summaries for text. In this tutorial, I will be covering an extractive method in which the top 2 most important sentences in an article serve as the summary. This method uses GloVe, cosine similarity and TextRank.
GloVe is an algorithm which creates vector (or numerical) representations for words. Cosine similarity is used to generate a similarity matrix n x n (n = number of sentences in the article) and the TextRank Algorithm is used to score the sentence similarities. The top scoring (most similar) sentences are used to generate the summary.
Before feeding the data into the algorithm, it should be cleaned, preprocessed and converted into a numerical format that it can understand (vectorized).
Using Jupyter Notebook:
Step 1: Create a new environment or use an existing environment to create a new Jupyter Notebook. Make sure the environment that you are running your Jupyter Notebook in has the following dependencies installed:
dill=0.2.9
networkx=2.4
nltk=3.4.5
numpy=1.18.1
numpy-base=1.18.1
pandas=1.0.3
pickleshare=0.7.5
pymongo=3.8.0
python=3.7.7
regex=2018.07.11
scikit-learn=0.22.1Step 2: Download some needed files. This step only has to be done once.
Download and unzip the GloVe word vectors trained on Wikipedia and Gigaword 5 Data . This code applies to Jupyter Notebook.
!wget http://nlp.stanford.edu/data/glove.6B.zip
!unzip glove*.zipDownload nltk punkt
nltk.download('punkt')Step 3: Connect to the database collection. Here I am connecting to a MongoDB collection. Replace <> with the relevant information.
import re
from pymongo import MongoClient
#create a MongoClient to the running mongoDB instance (make sure that MongoDB instance is currently running)
#specify the host and port to connect on
MONGO_ADDRESS = "mongodb://<IP Address>"
from bson import json_util
from collections import Counter
def connection():
# Connect to the database
client = MongoClient(MONGO_ADDRESS)
db = client.<DBName>
# Handle to the connection
collection = db.<CollectionName>
data = list(collection.find({},{'<col1>','<col2>','<col3>','<col4>','<col5>','<col6>'}))
return data
data = connection()Step 4: Convert the collection into a dataframe. Pickle the file to be stored on your local machine. You only have to run this code once, or as often as the database updates.
import pandas as pd
df = pd.DataFrame.from_records(data)
#Converts dataframe to a file
df.to_pickle('filename.pkl')Step 5: Read the pickled file back into a pandas dataframe. Each time you run Jupyter Notebook, you only have to start from the code in this step (unless the MongoDB collection is updated).
import pandas as pd
df = pd.read_pickle('filename.pkl')Step 6: Now let’s do some data cleaning on a particular column.
Remove whitespace:
df['<column name>'] = df['<column name>'].str.replace(r'\s+', ' ')Remove chars \n and \t:
df['<column name>'] = df['<column name>'].str.replace(r'\\n', ' ')
df['<column name>'] = df['<column name>'].str.replace(r'\\t', ' ')Next, let’s identify the number of non-empty rows for the target column:
len(list(filter(lambda x: not x, df['<column name>'] == '')))Finally, the rows which are empty for the target column will be filtered out:
df = df[df['<column name>'] != '']Let’s confirm that we now have the same number of non-empty rows as shown above:
len(df)Remove non-English characters from text:
df['<column name>'] = df['<column name>'].str.replace(r'[^\x00-\x7F]+', '')Remove some comment code, anything between and including <!– and –>:
df['<column name>'] = df['<column name>'].str.replace("<!--.*-->","")Return only the rows which do not have NaN in <column name> column:
df = df[df['<column name>'].notna()]Step 7: Create the text summarizer function.
import nltk
nltk.download('punkt') # one time execution
# Use the sent_tokenize() function of the nltk library to break the text into individual sentences
from nltk.tokenize import sent_tokenize
nltk.download('stopwords')
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
from nltk.cluster.util import cosine_distance
import pandas as pd
import networkx as nx
# Define the summarizer function
def summarize_func(description):
sentences = []
sentences.append(sent_tokenize(description))
# Flatten list
sentences = [y for x in sentences for y in x]
# Extract word vectors
word_embeddings = {}
f = open('glove.6B.100d.txt', encoding='utf-8')
for line in f:
values = line.split()
word = values[0]
coefs = np.asarray(values[1:], dtype='float32')
word_embeddings[word] = coefs
f.close()
# Remove punctuations, numbers and special characters
clean_sentences = pd.Series(sentences).str.replace("[^a-zA-Z]", " ")
# Make the text lowercase
clean_sentences = [s.lower() for s in clean_sentences]
# Define function to remove stopwords
def remove_stopwords(sen):
sen_new = " ".join([i for i in sen if i not in stop_words])
return sen_new
# Remove stopwords from the sentences
clean_sentences = [remove_stopwords(r.split()) for r in clean_sentences]
# Vectorize each word in a sentence and then take the mean of those vectors to get a consolidated vector for the sentence
sentence_vectors = []
for i in clean_sentences:
if len(i) != 0:
v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()])/(len(i.split())+0.001)
else:
v = np.zeros((100,))
sentence_vectors.append(v)
# Create empty similarity matrix of dimensions n * n where n is the number of sentences
sim_mat = np.zeros([len(sentences), len(sentences)])
# Fill matrix with cosine similarity scores
for i in range(len(sentences)):
for j in range(len(sentences)):
if i != j:
sim_mat[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,100), sentence_vectors[j].reshape(1,100))[0,0]
# Convert the similarity matrix into a graph
nx_graph = nx.from_numpy_array(sim_mat)
# Apply the TextRank algorithm on this graph to get the sentence rankings
scores = nx.pagerank(nx_graph)
ranked_sentences = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)
# Extract the top 2 sentences as the summary
# If the article <column name> has only 1 sentence, return that sentence
if len(sent_tokenize(description)) == 1:
return sentences
else:
return ranked_sentences[0][1] + ' ' + ranked_sentences[1][1]Step 8: Create a new “Summary” column which contains the results of applying the summary function to the first 10 rows of a particular column within the dataframe.
first_ten_df = df.head(10)
first_ten_df["Summary"] = first_ten_df.apply(lambda row: summarize_func(row['<column name>']), axis=1)
first_ten_dfStep 9: View the dataframe with the new column “Summary”.
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', None)
first_ten_df[['Summary']]Step 10: Finally, write the dataframe to a csv file.
first_ten_df.to_csv(r'summarized_text.csv', index = False)Using PyCharm:
Step 1: Create an environment (using conda, etc.). Install the following dependencies in the environment:
dill=0.2.9
networkx=2.4
nltk=3.4.5
numpy=1.18.1
numpy-base=1.18.1
pandas=1.0.3
pickleshare=0.7.5
pymongo=3.8.0
python=3.7.7
regex=2018.07.11
scikit-learn=0.22.1Step 2: In PyCharm, go to File > Settings and add the interpreter (python.exe) within that environment. This will be necessary for running the code later.
Step 3: Create a module called clean.py which stores the function for cleaning each row of <column name> in the df dataframe. Any word such as ‘data’ can be passed in, as the variable the function is applied on will replace ‘data’.
def clean(data):
"""
Cleans data
"""
# Import statements
import re
import string
# Remove whitespace
data['<column name>'] = data['<column name>'].str.replace(r'\s+', ' ')
# Remove chars \n and \t
data['<column name>'] = data['<column name>'].str.replace(r'\\n', ' ')
data['<column name>'] = data['<column name>'].str.replace(r'\\t', ' ')
# Remove rows with Empty Descriptions
data = data[data['<column name>'] != '']
# Remove non-English characters from description
data['<column name>'] = data['<column name>'].str.replace(r'[^\x00-\x7F]+', '')
# Remove comment code
data['<column name>'] = data['<column name>'].str.replace("<!--.*-->", "")
# Return only the rows which do not have NaN in <column name> column:
data = data[data['<column name>'].notna()]
return dataStep 4: Create a module called summarizer.py which stores the setup and summarization functions which will later be called in main.py. Make sure it is stored in the same directory as clean.py.
The functions starting with ‘_’ are used by the setup and summarization functions, they will not be called in main.py.
# Import statements
import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import stopwords
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
from nltk.cluster.util import cosine_distance
import pandas as pd
import networkx as nx
import urllib.request
import zipfile
from os import path
def _download_file(url, save_path):
with urllib.request.urlopen(url) as dl_file:
with open(save_path, 'wb') as out_file:
out_file.write(dl_file.read())
def setup():
"""
Get GloVe word vectorizer if doesn't exist and downloads nltk dependencies
"""
if not path.exists('glove.6B.100d.txt'):
_download_file('http://nlp.stanford.edu/data/glove.6B.zip', 'glove6B.zip')
with zipfile.ZipFile('glove6B.zip', 'r') as zip_ref:
zip_ref.extractall('.')
# Run only once
nltk.download('punkt')
nltk.download('stopwords')
def _remove_stopwords(sen):
"""
Function to remove stopwords
"""
stop_words = stopwords.words('english')
sen_new = " ".join([i for i in sen if i not in stop_words])
return sen_new
def summarize(article_text):
"""
Summarizer function
:param article_text: Full article text for summarization
:return:
"""
sentences = []
sentences.append(sent_tokenize(article_text))
# Flatten list
sentences = [y for x in sentences for y in x]
# Extract word vectors
word_embeddings = {}
f = open('glove.6B.100d.txt', encoding='utf-8')
for line in f:
values = line.split()
word = values[0]
coefs = np.asarray(values[1:], dtype='float32')
word_embeddings[word] = coefs
f.close()
# Remove punctuations, numbers and special characters
clean_sentences = pd.Series(sentences).str.replace("[^a-zA-Z]", " ")
# Make text lowercase
clean_sentences = [s.lower() for s in clean_sentences]
# Remove stopwords from the sentences
clean_sentences = [_remove_stopwords(r.split()) for r in clean_sentences]
# Create consolidated vector for each sentence
sentence_vectors = []
for i in clean_sentences:
if len(i) != 0:
v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()]) / (len(i.split()) + 0.001)
else:
v = np.zeros((100,))
sentence_vectors.append(v)
# Create empty similarity matrix of dimensions n * n where n is the number of sentences
sim_mat = np.zeros([len(sentences), len(sentences)])
# Fill matrix with cosine similarity scores
for i in range(len(sentences)):
for j in range(len(sentences)):
if i != j:
sim_mat[i][j] = \
cosine_similarity(sentence_vectors[i].reshape(1, 100), sentence_vectors[j].reshape(1, 100))[0, 0]
# Convert the similarity matrix into a graph
nx_graph = nx.from_numpy_array(sim_mat)
# Apply the PageRank algorithm on this graph to get the sentence rankings
scores = nx.pagerank(nx_graph)
ranked_sentences = sorted(((scores[i], s) for i, s in enumerate(sentences)), reverse=True)
# Extract the top 2 sentences as the summary
# If the article <column name> has only 1 sentence, return that sentence
if len(sent_tokenize(article_text)) == 1:
return sentences
else:
return ranked_sentences[0][1] + ' ' + ranked_sentences[1][1]Step 5: Create a module main.py. Make sure it is stored in the same directory as clean.py and summarizer.py. It first reads in the database collection and stores the first 10 rows as a dataframe df. Then it calls the cleaning function from clean.py. Next it calls the setup function from summarizer.py. Finally, it calls the summarize function from summarizer.py and applies it to each row of <column name> in the dataframe df.
# Import statements
# Note: If you just use 'import summarizer', you would have to call it
# using 'summarizer.summarize' instead of just 'summarize'
from clean import clean
from summarizer import summarize
import pandas as pd
import re
from pymongo import MongoClient
if __name__ == "__main__":
# Read-in data from MongoDB collection
#specify the host and port to connect on
MONGO_ADDRESS = "mongodb://<IP Address>"
def connection():
# Connect to the database
client = MongoClient(MONGO_ADDRESS)
db = client.<DBName>
# Handle to the connection
collection = db.<CollectionName>
data = list(collection.find({},{'<col1>','<col2>','<col3>','<col4>','<col5>','<col6>'}))
return data
data = connection()
# Convert 'data' to pandas dataframe called 'df'
df = pd.DataFrame.from_records(data)
# Set 'df' variable as first 10 rows of 'df'
df = df.head(10)
# Calls clean function in clean.py to clean 'df' dataframe
df = clean(data)
# Calls setup() function in summarizer.py
summarizer.setup()
# Applies summarize() function in summarizer.py on each row of df under <column name>
df['Summary'] = df.apply(lambda row: summarize(row['<column name>']), axis=1)Step 6: You can check if the code in your summarizer.py module works by running it on some sample text. First set the script path to summarizer.py in Run > Edit Configurations. Then add the test code to the bottom of summarizer.py. Finally click Run > Run ‘summarizer’.
Alternatively, you can add the test code to the bottom of summarizer.py and then click the green play button to the left of if __name__ == “__main__” in summarizer.py.
# For testing
if __name__ == "__main__":
print("Setting up")
setup()
print("Setup complete")
article = """Insert article here."""
print("Summarizing")
print("Summary: " + summarize(article))Step 7: Finally, to summarize each row of the target column in the MongoDB collection and then add them to a new column called ‘Summary’:, first set the script path to main.py in Run > Edit Configurations. Then, click Run > Run ‘main’.
You can instead click on the green play button to the left of if __name__ == “__main__” in main.py.