There are several methods of generating summaries for text. In this tutorial, I will be covering an extractive method in which the top 2 most important sentences in an article serve as the summary. This method uses GloVe, cosine similarity and TextRank.

GloVe is an algorithm which creates vector (or numerical) representations for words. Cosine similarity is used to generate a similarity matrix n x n (n = number of sentences in the article) and the TextRank Algorithm is used to score the sentence similarities. The top scoring (most similar) sentences are used to generate the summary.

Before feeding the data into the algorithm, it should be cleaned, preprocessed and converted into a numerical format that it can understand (vectorized).

Using Jupyter Notebook:

Step 1: Create a new environment or use an existing environment to create a new Jupyter Notebook. Make sure the environment that you are running your Jupyter Notebook in has the following dependencies installed:

dill=0.2.9
networkx=2.4
nltk=3.4.5
numpy=1.18.1
numpy-base=1.18.1
pandas=1.0.3
pickleshare=0.7.5
pymongo=3.8.0
python=3.7.7
regex=2018.07.11
scikit-learn=0.22.1

Step 2: Download some needed files. This step only has to be done once.

Download and unzip the GloVe word vectors trained on Wikipedia and Gigaword 5 Data . This code applies to Jupyter Notebook.

!wget http://nlp.stanford.edu/data/glove.6B.zip
!unzip glove*.zip

Download nltk punkt

nltk.download('punkt')

Step 3: Connect to the database collection. Here I am connecting to a MongoDB collection. Replace <> with the relevant information.

import re

from pymongo import MongoClient

#create a MongoClient to the running mongoDB instance (make sure that MongoDB instance is currently running)

#specify the host and port to connect on
MONGO_ADDRESS = "mongodb://<IP Address>"
from bson import json_util
from collections import Counter

def connection():
    # Connect to the database
    client = MongoClient(MONGO_ADDRESS)
    db = client.<DBName>
    # Handle to the connection
    collection = db.<CollectionName>

    data = list(collection.find({},{'<col1>','<col2>','<col3>','<col4>','<col5>','<col6>'}))
        
    return data

data = connection()

Step 4: Convert the collection into a dataframe. Pickle the file to be stored on your local machine. You only have to run this code once, or as often as the database updates.

import pandas as pd

df = pd.DataFrame.from_records(data)

#Converts dataframe to a file
df.to_pickle('filename.pkl')

Step 5: Read the pickled file back into a pandas dataframe. Each time you run Jupyter Notebook, you only have to start from the code in this step (unless the MongoDB collection is updated).

import pandas as pd

df = pd.read_pickle('filename.pkl')

Step 6: Now let’s do some data cleaning on a particular column.

Remove whitespace:

df['<column name>'] = df['<column name>'].str.replace(r'\s+', ' ')

Remove chars \n and \t:

df['<column name>'] = df['<column name>'].str.replace(r'\\n', ' ')
df['<column name>'] = df['<column name>'].str.replace(r'\\t', ' ')

Next, let’s identify the number of non-empty rows for the target column:

len(list(filter(lambda x: not x, df['<column name>'] == '')))

Finally, the rows which are empty for the target column will be filtered out:

df = df[df['<column name>'] != '']

Let’s confirm that we now have the same number of non-empty rows as shown above:

len(df)

Remove non-English characters from text:

df['<column name>'] = df['<column name>'].str.replace(r'[^\x00-\x7F]+', '')

Remove some comment code, anything between and including <!– and –>:

df['<column name>'] = df['<column name>'].str.replace("<!--.*-->","")

Return only the rows which do not have NaN in <column name> column:

df = df[df['<column name>'].notna()]

Step 7: Create the text summarizer function.

import nltk

nltk.download('punkt') # one time execution

# Use the sent_tokenize() function of the nltk library to break the text into individual sentences
from nltk.tokenize import sent_tokenize

nltk.download('stopwords')

from nltk.corpus import stopwords

stop_words = stopwords.words('english')

from sklearn.metrics.pairwise import cosine_similarity

import numpy as np

from nltk.cluster.util import cosine_distance

import pandas as pd

import networkx as nx

# Define the summarizer function
def summarize_func(description):
    sentences = []
    sentences.append(sent_tokenize(description))

    # Flatten list
    sentences = [y for x in sentences for y in x] 

    # Extract word vectors
    word_embeddings = {}
    f = open('glove.6B.100d.txt', encoding='utf-8')
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        word_embeddings[word] = coefs
    f.close()

    # Remove punctuations, numbers and special characters
    clean_sentences = pd.Series(sentences).str.replace("[^a-zA-Z]", " ")

    # Make the text lowercase
    clean_sentences = [s.lower() for s in clean_sentences]

    # Define function to remove stopwords
    def remove_stopwords(sen):
        sen_new = " ".join([i for i in sen if i not in stop_words])
        return sen_new

    # Remove stopwords from the sentences
    clean_sentences = [remove_stopwords(r.split()) for r in clean_sentences]

    # Vectorize each word in a sentence and then take the mean of those vectors to get a consolidated vector for the sentence
    sentence_vectors = []
    for i in clean_sentences:
        if len(i) != 0:
            v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()])/(len(i.split())+0.001)
        else:
            v = np.zeros((100,))
        sentence_vectors.append(v)

    # Create empty similarity matrix of dimensions n * n where n is the number of sentences
    sim_mat = np.zeros([len(sentences), len(sentences)])

    # Fill matrix with cosine similarity scores
    for i in range(len(sentences)):
        for j in range(len(sentences)):
            if i != j:
                sim_mat[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,100), sentence_vectors[j].reshape(1,100))[0,0]

    # Convert the similarity matrix into a graph
    nx_graph = nx.from_numpy_array(sim_mat)

    # Apply the TextRank algorithm on this graph to get the sentence rankings
    scores = nx.pagerank(nx_graph)

    ranked_sentences = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)

    # Extract the top 2 sentences as the summary
    # If the article <column name> has only 1 sentence, return that sentence
    if len(sent_tokenize(description)) == 1:
        return sentences
    else:
        return ranked_sentences[0][1] + ' ' + ranked_sentences[1][1]

Step 8: Create a new “Summary” column which contains the results of applying the summary function to the first 10 rows of a particular column within the dataframe.

first_ten_df = df.head(10)

first_ten_df["Summary"] = first_ten_df.apply(lambda row: summarize_func(row['<column name>']), axis=1)

first_ten_df

Step 9: View the dataframe with the new column “Summary”.

pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', None)

first_ten_df[['Summary']]

Step 10: Finally, write the dataframe to a csv file.

first_ten_df.to_csv(r'summarized_text.csv', index = False)

Using PyCharm:

Step 1: Create an environment (using conda, etc.). Install the following dependencies in the environment:

dill=0.2.9
networkx=2.4
nltk=3.4.5
numpy=1.18.1
numpy-base=1.18.1
pandas=1.0.3
pickleshare=0.7.5
pymongo=3.8.0
python=3.7.7
regex=2018.07.11
scikit-learn=0.22.1

Step 2: In PyCharm, go to File > Settings and add the interpreter (python.exe) within that environment. This will be necessary for running the code later.

Step 3: Create a module called clean.py which stores the function for cleaning each row of <column name> in the df dataframe. Any word such as ‘data’ can be passed in, as the variable the function is applied on will replace ‘data’.

def clean(data):
    """
    Cleans data
    """

    # Import statements
    import re
    import string

    # Remove whitespace
    data['<column name>'] = data['<column name>'].str.replace(r'\s+', ' ')

    # Remove chars \n and \t
    data['<column name>'] = data['<column name>'].str.replace(r'\\n', ' ')
    data['<column name>'] = data['<column name>'].str.replace(r'\\t', ' ')

    # Remove rows with Empty Descriptions
    data = data[data['<column name>'] != '']

    # Remove non-English characters from description
    data['<column name>'] = data['<column name>'].str.replace(r'[^\x00-\x7F]+', '')

    # Remove comment code
    data['<column name>'] = data['<column name>'].str.replace("<!--.*-->", "")

    # Return only the rows which do not have NaN in <column name> column:
    data = data[data['<column name>'].notna()]

    return data

Step 4: Create a module called summarizer.py which stores the setup and summarization functions which will later be called in main.py. Make sure it is stored in the same directory as clean.py.

The functions starting with ‘_’ are used by the setup and summarization functions, they will not be called in main.py.

# Import statements
import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import stopwords
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
from nltk.cluster.util import cosine_distance
import pandas as pd
import networkx as nx
import urllib.request
import zipfile
from os import path


def _download_file(url, save_path):
    with urllib.request.urlopen(url) as dl_file:
        with open(save_path, 'wb') as out_file:
            out_file.write(dl_file.read())


def setup():
    """
    Get GloVe word vectorizer if doesn't exist and downloads nltk dependencies
    """
    if not path.exists('glove.6B.100d.txt'):
        _download_file('http://nlp.stanford.edu/data/glove.6B.zip', 'glove6B.zip')
        with zipfile.ZipFile('glove6B.zip', 'r') as zip_ref:
            zip_ref.extractall('.')

    # Run only once
    nltk.download('punkt')
    nltk.download('stopwords')


def _remove_stopwords(sen):
    """
    Function to remove stopwords
    """
    stop_words = stopwords.words('english')
    sen_new = " ".join([i for i in sen if i not in stop_words])
    return sen_new


def summarize(article_text):
    """
    Summarizer function
    :param article_text: Full article text for summarization
    :return:
    """
    sentences = []
    sentences.append(sent_tokenize(article_text))

    # Flatten list
    sentences = [y for x in sentences for y in x]

    # Extract word vectors
    word_embeddings = {}
    f = open('glove.6B.100d.txt', encoding='utf-8')
    for line in f:
        values = line.split()
        word = values[0]
        coefs = np.asarray(values[1:], dtype='float32')
        word_embeddings[word] = coefs
    f.close()

    # Remove punctuations, numbers and special characters
    clean_sentences = pd.Series(sentences).str.replace("[^a-zA-Z]", " ")

    # Make text lowercase
    clean_sentences = [s.lower() for s in clean_sentences]

    # Remove stopwords from the sentences
    clean_sentences = [_remove_stopwords(r.split()) for r in clean_sentences]

    # Create consolidated vector for each sentence
    sentence_vectors = []
    for i in clean_sentences:
        if len(i) != 0:
            v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()]) / (len(i.split()) + 0.001)
        else:
            v = np.zeros((100,))
        sentence_vectors.append(v)

    # Create empty similarity matrix of dimensions n * n where n is the number of sentences
    sim_mat = np.zeros([len(sentences), len(sentences)])

    # Fill matrix with cosine similarity scores
    for i in range(len(sentences)):
        for j in range(len(sentences)):
            if i != j:
                sim_mat[i][j] = \
                    cosine_similarity(sentence_vectors[i].reshape(1, 100), sentence_vectors[j].reshape(1, 100))[0, 0]

    # Convert the similarity matrix into a graph
    nx_graph = nx.from_numpy_array(sim_mat)

    # Apply the PageRank algorithm on this graph to get the sentence rankings
    scores = nx.pagerank(nx_graph)

    ranked_sentences = sorted(((scores[i], s) for i, s in enumerate(sentences)), reverse=True)

    # Extract the top 2 sentences as the summary
    # If the article <column name> has only 1 sentence, return that sentence
    if len(sent_tokenize(article_text)) == 1:
        return sentences
    else:
        return ranked_sentences[0][1] + ' ' + ranked_sentences[1][1]

Step 5: Create a module main.py. Make sure it is stored in the same directory as clean.py and summarizer.py. It first reads in the database collection and stores the first 10 rows as a dataframe df. Then it calls the cleaning function from clean.py. Next it calls the setup function from summarizer.py. Finally, it calls the summarize function from summarizer.py and applies it to each row of <column name> in the dataframe df.

# Import statements
# Note: If you just use 'import summarizer', you would have to call it
# using 'summarizer.summarize' instead of just 'summarize'
from clean import clean
from summarizer import summarize
import pandas as pd
import re
from pymongo import MongoClient

if __name__ == "__main__":

     # Read-in data from MongoDB collection

     #specify the host and port to connect on
     MONGO_ADDRESS = "mongodb://<IP Address>"

     def connection():
          # Connect to the database
          client = MongoClient(MONGO_ADDRESS)
          db = client.<DBName>
          # Handle to the connection
          collection = db.<CollectionName>

          data = list(collection.find({},{'<col1>','<col2>','<col3>','<col4>','<col5>','<col6>'}))
        
          return data

     data = connection()

     # Convert 'data' to pandas dataframe called 'df'
     df = pd.DataFrame.from_records(data)

     # Set 'df' variable as first 10 rows of 'df'
     df = df.head(10)

     # Calls clean function in clean.py to clean 'df' dataframe
     df = clean(data)

     # Calls setup() function in summarizer.py
     summarizer.setup()

     # Applies summarize() function in summarizer.py on each row of df under <column name>
     df['Summary'] = df.apply(lambda row: summarize(row['<column name>']), axis=1)

Step 6: You can check if the code in your summarizer.py module works by running it on some sample text. First set the script path to summarizer.py in Run > Edit Configurations. Then add the test code to the bottom of summarizer.py. Finally click Run > Run ‘summarizer’.

Alternatively, you can add the test code to the bottom of summarizer.py and then click the green play button to the left of if __name__ == “__main__” in summarizer.py.

# For testing
if __name__ == "__main__":
    print("Setting up")
    setup()

    print("Setup complete")

    article = """Insert article here."""

    print("Summarizing")
    print("Summary: " + summarize(article))

Step 7: Finally, to summarize each row of the target column in the MongoDB collection and then add them to a new column called ‘Summary’:, first set the script path to main.py in Run > Edit Configurations. Then, click Run > Run ‘main’.

You can instead click on the green play button to the left of if __name__ == “__main__” in main.py.