Text summarization is useful for highlighting crucial information for business such as buying signals. There are some simple methods of summarizing text, although less effective than using deep learning. In this tutorial we will be using the Heap queue algorithm instead of a neural network to create a summary.

To replicate these steps, you need to have the following installed: Python, Beautiful Soup, spaCy, and spaCy en_core_web_sm.

Step 1: Scrape the data from the webpage using Beautiful Soup. Since I had to specify a list of paragraphs to be scraped, I had to export then reimport the scraped data in order to convert it to text.

#import the Beautiful Soup module
from bs4 import BeautifulSoup

#Import the request attribute of the urllib module
import urllib.request

#Add URL of website to be scraped
quote_page = 'https://www.businesswire.com/news/home/20200319005385/en/T-Mobile-Financially-Prepared-Close-Sprint-Merger'

#Query the website and store the code in the variable page
page = urllib.request.urlopen(quote_page)

#Parse the HTML code using Beautiful Soup and store it as the variable soup
soup = BeautifulSoup(page, 'html.parser')

#Declare the variable f to open a TXT file called text
#w represents write, the kind of permission/operation to be done on the file
#set encoding to UTF to avoid errors for special characters
f= open('../text.txt','w+', encoding='utf-8')

#Get relevant section from webpage and write to file
html = soup.find('div', class_='bw-release-story').findAll('p')[0:5]
for text in html:
    f.write(str(text))
f.close()

#Import file
f=open('../text.txt', 'r', encoding='utf8')
if f.mode == 'r':
    text = f.read()

#Convert to text
text = BeautifulSoup(text).text

Step 2: Convert each word into a token and clean the text by removing stop words and punctuation. Stop words are very commonly used and deemed unimportant for the purposes of data mining and NLP. Also, the frequency, maximum frequency and normalized frequency is calculated per token.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
from string import punctuation

stopwords = list(STOP_WORDS)

nlp = spacy.load('en_core_web_sm')

#Get tokenized words in doc
doc = nlp(text)

#Create list of tokens
tokens = [token.text for token in doc]
print(tokens)

#Add "new line" to punctuation
punctuation = punctuation + '\n'
punctuation

#Calculate how many times each token appears in text, increasing frequency 
#for each time it appears
#Only calculate for tokens that are not stop words or punctuation
word_frequencies = {}
for word in doc:
    if word.text.lower() not in stopwords:
        if word.text.lower() not in punctuation:
            if word.text not in word_frequencies.keys():
                word_frequencies[word.text] = 1
            else:
                word_frequencies[word.text] += 1

print(word_frequencies)

max_frequency = max(word_frequencies.values())

#Determine normalized frequency
for word in word_frequencies.keys():
    word_frequencies[word] = word_frequencies[word]/max_frequency 

Step 3: The sentences are tokenized and then scored. The normalized frequency of each word in a sentence is combined. The sentence with the highest combined frequencies is the highest scorer. The number of the top 30% highest scoring sentences is acertained.

#Tokenize sentences
sentence_tokens = [sent for sent in doc.sents]
print(sentence_tokens)

#Calculate sentence scores: add normalized frequency of each word token for 
#sentence, to pick most important sentences
sentence_scores = {}
for sent in sentence_tokens:
    for word in sent:
        if word.text.lower() in word_frequencies.keys():
            if sent not in sentence_scores.keys():
                sentence_scores[sent] = word_frequencies[word.text.lower()]
            else:
                sentence_scores[sent] += word_frequencies[word.text.lower()]

#Show sentence score for each sentence
sentence_scores

from heapq import nlargest

#Select number of top 30% highest scoring sentences
select_length = int(len(sentence_tokens)*0.3)

#Select select_length number of sentences that have max word tokens frequency count
select_length

Step 4: We get the summary by determining the 30% (n) largest word tokens from our sentence tokens.

summary = nlargest(select_length, sentence_scores, key = sentence_scores.get)

Step 5: Combine the sentences into a list, join the elements of that list and obtain the final summary!

#Combine the sentences together into list
final_summary = [word.text for word in summary]

#Join the elements of the list
summary = ' '.join(final_summary)

#The final summary!
print(summary)

You can see that the summary text is approximately 46.8% shorter than the original text:

#Length of the original text
len(text)

#Length of the summary
len(summary)

#The percentage that the text was shortened in creating the summary
percentage_shortened = ((len(summary))/(len(text)))*100
percentage_shortened

Here is the resulting summary:

today announced that it is currently financially prepared to close its planned merger with Sprint, based on its previously secured commitments for bridge financing and senior credit facility financing, subject to the satisfaction of customary closing conditions under its commitment letter. This diversification of banks, and the spreading of the committed bridge financing creates a very high-quality bridge,” said Braxton Carter, Chief Financial Officer of T-Mobile.

 “In times when consumers need affordable service plans to stay connected, T-Mobile is fully prepared and well positioned to be the provider to meet these needs,” said Mike Sievert, President and Chief Operating Officer of T-Mobile. The company has been in communication with all sixteen banks and has not received any notification that any of the banks are unprepared to fund their commitments to support the closing of the merger transaction. We can see the finish line and are prepared to close the merger very soon so our teams can get to work building a supercharged Un-carrier.”