Named Entity Recognition involves identifying portions of text representing labels such as geographical location, geopolitical entity, persons, etc. In this tutorial, we will use deep learning to identify various entities in Medium articles and present them in useful way.
In Jupyter Notebook:
Here are some of the dependencies used:
- jupyter=1.0.0
- jupyter_client=6.1.2
- jupyter_console=6.1.0
- jupyter_core=4.6.3
- keras=2.1.6
- keras-applications=1.0.8
- keras-preprocessing=1.1.0
- matplotlib=3.2.1=0
- matplotlib-base=3.2.1
- nltk==3.4.5
- numpy==1.18.2
- pandas=1.0.3
- pickle-mixin==1.0.2
- python=3.6.10
- scikit-learn=0.22.2.post1
- scipy==1.4.1
- sklearn==0.0
- tensorboard==1.8.0
- tensorflow==1.8.0
- tensorflow-estimator==2.1.0
- tensorflow-hub==0.3.0Training the model
Step 1: The dataset we will use for training the model can be found here. Create a new environment or use an existing environment to create a new Jupyter Notebook. In the Notebook, read in as many sentences of the ner_dataset as your computer can handle. I chose 6000, so I read in 131419 rows of the dataset.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.style.use("ggplot")
data = pd.read_csv("ner_dataset.csv", encoding="latin1")
# Read in portion of dataset with 6000 sentences
sample_size = 131419
data = data.head(sample_size)
data = data.fillna(method="ffill")
data.tail(10)words = list(set(data["Word"].values))
words.append("ENDPAD")
n_words = len(words); n_wordstags = list(set(data["Tag"].values))
n_tags = len(tags); n_tagsStep 2: Use the SentenceGetter class to retrieve sentences with their labels.
class SentenceGetter(object):
def __init__(self, data):
self.n_sent = 1
self.data = data
self.empty = False
agg_func = lambda s: [(w, p, t) for w, p, t in zip(s["Word"].values.tolist(),
s["POS"].values.tolist(),
s["Tag"].values.tolist())]
self.grouped = self.data.groupby("Sentence #").apply(agg_func)
self.sentences = [s for s in self.grouped]
def get_next(self):
try:
s = self.grouped["Sentence: {}".format(self.n_sent)]
self.n_sent += 1
return s
except:
return Nonegetter = SentenceGetter(data)sent = getter.get_next()This is how a sentence looks now.
print(sent)Step 3: Get all sentences
sentences = getter.sentencesStep 4: Set max sentence length to 50. Create dictionary of tags to map labels to numbers.
max_len = 50
tag2idx = {t: i for i, t in enumerate(tags)}Step 5: Split the ‘Word’ portion of the sampled dataset into a list of sentences each containing a list of words.
X = [[w[0] for w in s] for s in sentences]Step 6: Add padding to each list of words in sentence list until the len of the word list is 50.
new_X = []
for seq in X:
new_seq = []
for i in range(max_len):
try:
new_seq.append(seq[i])
except:
new_seq.append("__PAD__")
new_X.append(new_seq)
X = new_XStep 7: Also split the tags into lists within lists.
y = [[tag2idx[w[2]] for w in s] for s in sentences]Step 8: Pad the tags as well
from keras.preprocessing.sequence import pad_sequences
y = pad_sequences(maxlen=max_len, sequences=y, padding="post", value=tag2idx["O"])Step 9: Set batch size.
batch_size = 50Step 10: Initialize the ELMo embedding from tensorflow hub.
import tensorflow as tf
import tensorflow_hub as hub
from keras import backend as KStep 11: Initialize the tensorflow session.
sess = tf.Session()
K.set_session(sess)Step 12: Download bi-directional LSTM model with ELMo word embeddings. ELMo word embeddings allow the model to learn both word (e.g., syntax and semantics) and linguistic context of a training dataset.
elmo_model = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())If you prefer to download the model from here and load it from your computer (in case the website is down): save “saved_model.pb” and “tfhub_module.pb” in a folder called “9bb74bc86f9caffc8c47dd7b33ec4bb354d9602d.” Make sure to create two new folders called “assets” and “variables” within the folder. Then change “https://tfhub.dev/google/elmo/2” to the long folder name.
Step 13: Create a function that vectorizes a sequence of strings with the ELMo embedding.
def ElmoEmbedding(x):
return elmo_model(inputs={
"tokens": tf.squeeze(tf.cast(x, tf.string)),
"sequence_len": tf.constant(batch_size*[max_len])
},
signature="tokens",
as_dict=True)["elmo"]Step 14: Create a residual LSTM network with an ELMo embedding layer.
from keras.models import Model, Input
from keras.layers.merge import add
from keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional, Lambdainput_text = Input(shape=(max_len,), dtype=tf.string)
embedding = Lambda(ElmoEmbedding, output_shape=(max_len, 1024))(input_text)
x = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(embedding)
x_rnn = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(x)
x = add([x, x_rnn]) # residual connection to the first biLSTM
out = TimeDistributed(Dense(n_tags, activation="softmax"))(x)model = Model(input_text, out)model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])Step 15: In this step I set the training and test data. 80% of the 6,000 total sentences (contained in the variable X) of the sampled dataset will be used to train the model. The remaining 20% will be used to validate the model.
If you prefer to use a different number of sentences (other than 6,000) for training and validation, make sure you choose a sample_size that contains your prefered number of sentences.
Also make sure that the number of sentences you select, train_size and val_size are each divisible by the batch_size. Feel free to alter any of these parameters, the batch_size or train/test split proportion to achieve this.
num_sentences = len(X)
train_size = int(num_sentences * .8)
val_size = num_sentences - train_size
X_tr, X_val = X[:train_size], X[-val_size:]
y_tr, y_val = y[:train_size], y[-val_size:]
#The input layer of NN is in 3D so reshape makes labels 3D
y_tr = y_tr.reshape(y_tr.shape[0], y_tr.shape[1], 1)
y_val = y_val.reshape(y_val.shape[0], y_val.shape[1], 1)Step 16: Train the model. The model will learn the relationships between the dependent variables (the outcome, what is being predicted) and independent variables.
At the end of the first epoch, the following error may be thrown: “InvalidArgumentError (see above for traceback): assertion failed: [Expected shape for Tensor sequence_length:0 is ] [?] [ but saw shape: ] [50]”
If you see this error, change the batch_size to the number represented by ?. Then, train the model again.
history = model.fit(np.array(X_tr), y_tr, validation_data=(np.array(X_val), y_val),
batch_size=batch_size, epochs=5, verbose=1)Step 17: Plot. Accuracy refers to how well the model makes predictions on the training sets. Validation accuracy refers to how well the model make predictions on the test sets. The model is overfitting if accuracy is high and validation accuracy is low. Can try to prevent overfitting by training with more data, using early stopping to stop model training before validation accuracy begins to drop off, etc.
hist = pd.DataFrame(history.history)plt.figure(figsize=(12,12))
plt.plot(hist["acc"])
plt.plot(hist["val_acc"])
plt.title("Learning curves")
plt.legend()
plt.show()Step 18: Save the model. This way you will not have to keep retraining the model each time you run Jupyter Notebook. The model weights are the different parameters used in each layer of the neural network.
model.save_weights('./testmodel_weights')
type(model)Step 19: Be sure to pickle the tags (tuples). If you need to reload the saved trained model, you will also need to reload the tags with which the model was trained.
import pickle
with open('tags.pickle', 'wb') as f:
pickle.dump(tags, f)When you are ready to use your trained model to make some predictions.
Step 1: Read in the data that you will be making predictions on. Let’s use this dataset containing titles, text, etc of Medium articles.
df = pd.read_csv("articles.csv", encoding="latin1")Step 2: Clean the dataset that the predictions will be made on. The column ‘text’ contains the content of each Medium article, so we will only need to clean that column. First examine the column to see which cleaning steps need to be taken. I applied the following changes:
df['text'] = df['text'].str.replace(r'\s+', ' ')df['text'] = df['text'].str.replace(r'\\n', ' ')
df['text'] = df['text'].str.replace(r'\\t', ' ')df = df[df['text'] != '']df['text'] = df['text'].str.replace(r'[^\x00-\x7F]+', '')df['text'] = df['text'].str.replace("<!--.*-->","")df = df[df['text'].notna()]Step 3: Load the pickled tags.
import pickle
tags = None
with open('tags.pickle', 'rb') as f:
tags = pickle.load(f)
n_tags = len(tags)Step 4: Read in some more of the code used in the training steps.
Start a tensorflow session.
import tensorflow as tf
import tensorflow_hub as hub
from keras import backend as Ksess = tf.Session()
K.set_session(sess)Download bi-directional LSTM model pretrained with ELMo word embeddings to learn both word (e.g., syntax and semantics) and linguistic context of a training dataset.
elmo_model = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())Create a function that vectorizes a sequence of strings with the ELMo embedding.
def ElmoEmbedding(x):
return elmo_model(inputs={
"tokens": tf.squeeze(tf.cast(x, tf.string)),
"sequence_len": tf.constant(batch_size*[max_len])
},
signature="tokens",
as_dict=True)["elmo"]Step 5: Set the max sentence length. It will be different than in the training script.
max_len = 200Step 6: Set batch size. It will be different than in the training script. It must be divisable by 32 because sequence_len is dtype=int32 and so needs to be made shape=(32,).
batch_size = 32Create a residual LSTM network with an ELMo embedding layer.
from keras.models import Model, Input
from keras.layers.merge import add
from keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional, Lambdainput_text = Input(shape=(max_len,), dtype=tf.string)
embedding = Lambda(ElmoEmbedding, output_shape=(max_len, 1024))(input_text)
x = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(embedding)
x_rnn = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(x)
x = add([x, x_rnn]) # residual connection to the first biLSTM
out = TimeDistributed(Dense(n_tags, activation="softmax"))(x)model = Model(input_text, out)model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])Step 7: Load in the trained model that you previously saved.
model.load_weights('./testmodel_weights')Step 8: Function which adds word padding to prediction dataset. Different from training script.
from nltk.tokenize import sent_tokenize, word_tokenize
from typing import List
def add_word_padding(words: List[str], words_per_sent = 200, pad_str = '__PAD__') -> List[str]:
num_pad_words = words_per_sent - len(words)
pad_words = ['__PAD__'] * num_pad_words
return words + pad_wordsStep 9: Function which adds sentence padding to prediction dataset. Each article is 512 sentences length. Both batch size and sent_per_article must be divisable by 32 because sequence_len is dtype=int32 and so needs to be made shape=(32,).
If you know that the max sentence length of each article is quite a bit below 512 words, you should definitely lower sent_per_article. Of course, the lower the sentence length, the faster the prediction will run. However, make sure that you keep sent_per_article divisable by 32, so if the longest sentence length is 300 words, change sent_per_article to the next number above that which is divisible by 32 (so, in that case, 320).
If you know that the max word length of each sentence of each article is say 100, lower words_per_sent to 100. There is no need for words_per_sent to be divisable by a particular number.
def gen_pad_sentences(sentences: List[List[str]], words_per_sent = 200, sent_per_article = 512, pad_str = '__PAD__') -> List[List[str]]:
pad_sent = ['__PAD__'] * words_per_sent
num_pad_sentences = sent_per_article - len(sentences)
pad_sentences = [pad_sent] * num_pad_sentences
return pad_sentencesStep 10: Preprocessing function which splits an article into a list of sentences, then splits each sentence into a list of words, and then applies word and sentence padding.
def create_nn_input(article: str) -> List:
# split article into list of sentences
sentences: List[str] = sent_tokenize(article)
# split sentences into list of words
sentences_split_into_words = [word_tokenize(sentence) for sentence in sentences]
# add padding to words
pad_words = [add_word_padding(sentence) for sentence in sentences_split_into_words]
# add padding to sentences
return pad_words + gen_pad_sentences(pad_words)Step 11: Predicting function
def predict_article(article):
#create list of lists
preprocessed_article = np.array(create_nn_input(article))
predictions = model.predict(preprocessed_article)
# picks best tag for each word
predicted_label_indices = np.argmax(predictions, axis=-1)
# gets tags of each element of a
convert_tags = np.vectorize(lambda x: tags[x])
#apply convert_tags on predicted_label_indices
predicted_tags = convert_tags(predicted_label_indices)
#todo: merge b and i pairs
#2d array for labels create pairwise tuples with 2d array of words
flat_words = preprocessed_article.flatten()
flat_tags = predicted_tags.flatten()
combine_tags_words = list(zip(flat_tags, flat_words))
#function, for each tuple (lbl, word) in result, keep tuples where label (t[0]) not O
return list(filter(lambda t: t[0] != 'O', combine_tags_words))Step 12: Apply predicting function on each text row in dataset.
prediction = df.apply(lambda row: predict_article(row['text']), axis=1)Step 13: Now you have made the predictions and they are in lists of tuples. Each tuple contains a word and tag pair. Each list of tuples corresponds to a different Medium article.
Next we will format the predictions and add them to the predicting dataset under a new column called “Entities”. Formatting them will allow us to more clearly see the entities extracted from each Medium article.
First convert prediction into a pandas dataframe:
prediction = pd.DataFrame(prediction)Make a bunch of functions that format the predictions
from itertools import groupbydef merge_tags(x):
# Change list of tuples into list
nested_list = [item for t in x for item in t]
# Add string 'split' before each tag containing B in 0th index of tag
res = []
for entry in nested_list:
if entry[0:2] == 'B-':
res.append('split')
res.append(entry)
nested_list[:] = res
res
# Split by string 'split'
def split_condition(x):
return x in {'split'}
grouper = groupby(res, key=split_condition)
# Convert to dictionary via enumerate
conv_to_dict = dict(enumerate((list(j) for i, j in grouper if not i), 1))
# Convert dictionary of lists into list of lists
dictionary_to_list = [[k]+v for k,v in conv_to_dict.items()]
# Remove first element in each list
for x in dictionary_to_list:
del x[0]
return dictionary_to_list# Apply merge_tags function to each list in list
def apply_merge_tags(x):
return [replace_B(l) for l in x]# Change tag type for B if group contains two I's, then drop the I's.
# Otherwise, just drop the I's.
def replace_B(x):
result = x
#get all I-'s
I_tags = list(filter(lambda x: x[:2] == 'I-', x))
if len(I_tags) == 2:
second_to_last_I_tag = I_tags[-2]
new_B_tag = 'B' + second_to_last_I_tag[1:]
bb_no_Is = list(filter(lambda x: x[:2] != 'I-', x))
result = bb_no_Is.copy()
result[0] = new_B_tag
else:
bb_no_Is = list(filter(lambda x: x[:2] != 'I-', x))
result = bb_no_Is.copy()
return result# Change B- tags with more readable label names
def change_label(label):
if label == 'B-geo':
return 'Location:'
elif label == 'B-gpe':
return 'Geopolitical Entity:'
elif label == 'B-org':
return 'Company:'
elif label == 'B-per':
return 'Person:'
elif label == 'B-tim':
return 'Time Period:'
else: return label# Apply change_label function to first element of each list
# Then change that element to show the result of applied function
def apply_change_label(y):
for x in y:
x[0] = change_label(x[0])# Join all but first element into string
def join_string(y):
return [[x[0]] + [' '.join(x[1:])] for x in y]# Merge lists with same first element and create dictionary
def merge_lists(x):
from collections import defaultdict
d = defaultdict(list)
for i, j in x:
d[i].append(j)
return dApply the functions to each article
prediction = prediction.apply(lambda row: merge_tags(row[0]), axis=1)# Convert prediction to df
prediction = pd.DataFrame(prediction)prediction = prediction.apply(lambda row: apply_merge_tags(row[0]), axis=1)# Convert prediction to df
prediction = pd.DataFrame(prediction)prediction = prediction.apply(lambda row: replace_B(row[0]), axis=1)# Convert prediction to df
prediction = pd.DataFrame(prediction)prediction = prediction.apply(lambda row: change_label(row[0]), axis=1)# Convert prediction to df
prediction = pd.DataFrame(prediction)prediction.apply(lambda row: apply_change_label(row[0]), axis=1)prediction = prediction.apply(lambda row: join_string(row[0]), axis=1)# Convert prediction to df
prediction = pd.DataFrame(prediction)Apply final function to prediction and apply it to new column in df called Entities
df["Entities"] = prediction.apply(lambda row: merge_lists(row[0]), axis=1)Now you have a dataset of Medium articles with a new Entities column containing the formatted entities for each article. These entities can be viewed on the front end by converting the entities (nested list format) into JSON objects.
In PyCharm:
To run the prediction code in PyCharm, you can convert it into a series of functions and store them in two new modules. Then, write code in main.py which calls the functions from the modules.
Step 1: Create an environment (using conda, etc.). Install the following dependencies in the environment:
- jupyter=1.0.0
- jupyter_client=6.1.2
- jupyter_console=6.1.0
- jupyter_core=4.6.3
- keras=2.1.6
- keras-applications=1.0.8
- keras-preprocessing=1.1.0
- matplotlib=3.2.1=0
- matplotlib-base=3.2.1
- nltk==3.4.5
- numpy==1.18.2
- pandas=1.0.3
- pickle-mixin==1.0.2
- python=3.6.10
- scikit-learn=0.22.2.post1
- scipy==1.4.1
- sklearn==0.0
- tensorboard==1.8.0
- tensorflow==1.8.0
- tensorflow-estimator==2.1.0
- tensorflow-hub==0.3.0Step 2: In PyCharm, go to File > Settings and add the interpreter (python.exe) within that environment. This will be necessary for running the code later.
Step 3: Create a module called clean.py which stores the function for cleaning each row of [‘text’] in the df dataframe. Any word such as ‘data’ can be passed in, as the variable the function is applied on will replace ‘data’.
def clean(data):
"""
Cleans data
"""
# Import statements
import re
import string
# Remove whitespace
data['text'] = data['text'].str.replace(r'\s+', ' ')
# Remove chars \n and \t
data['text'] = data['text'].str.replace(r'\\n', ' ')
data['text'] = data['text'].str.replace(r'\\t', ' ')
# Remove rows with Empty Descriptions
data = data[data['text'] != '']
# Remove non-English characters from description
data['text'] = data['text'].str.replace(r'[^\x00-\x7F]+', '')
# Remove comment code
data['text'] = data['text'].str.replace("<!--.*-->", "")
# Return only the rows which do not have NaN in ['text'] column:
data = data[data['text'].notna()]
return dataStep 4: In the same directory as clean.py, store the trained model and pickled tags from the Jupyter Notebook training steps.
Step 5: Create a module called entity_recognition.py which stores the setup and entity recognition functions which will later be called in main.py. Make sure it is stored in the same directory as clean.py.
The functions starting with ‘_’ are used by the setup and entity recognition functions, they will not be called in main.py.
# Import statements
import pickle
from keras.models import Model, Input
from keras.layers.merge import add
from keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional, Lambda
from keras import backend as K
import tensorflow as tf
import tensorflow_hub as hub
from itertools import groupby
from typing import List, Tuple, Dict
from collections import defaultdict
from nltk.tokenize import sent_tokenize, word_tokenize
import numpy as np
"""
Load in tags, set batch size, set max sentence length, create NN with elmo embedding layer.
Load in trained model.
"""
# Global variables
tags = None
n_tags = None
model = None
# Import tags
def _import_tags():
global tags
global n_tags
with open('tags.pickle', 'rb') as f:
tags = pickle.load(f)
n_tags = len(tags)
# Define model
def _define_model():
global model
sess = tf.Session()
K.set_session(sess)
elmo_model = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
def _elmo_embedding(x):
return elmo_model(inputs={
"tokens": tf.squeeze(tf.cast(x, tf.string)),
"sequence_len": tf.constant(batch_size * [max_len])
},
signature="tokens",
as_dict=True)["elmo"]
max_len = 200
batch_size = 32
input_text = Input(shape=(max_len,), dtype=tf.string)
embedding = Lambda(_elmo_embedding, output_shape=(max_len, 1024))(input_text)
x = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(embedding)
x_rnn = Bidirectional(LSTM(units=512, return_sequences=True,
recurrent_dropout=0.2, dropout=0.2))(x)
x = add([x, x_rnn]) # residual connection to the first biLSTM
out = TimeDistributed(Dense(n_tags, activation="softmax"))(x)
model = Model(input_text, out)
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=["accuracy"])
model.load_weights('./testmodel_weights')
# Setup function
def setup():
_import_tags()
_define_model()
"""
Tokenize and add word and sentence padding to prediction dataset.
Predict tags and make them readable.
"""
# Add word padding to article
def _add_word_padding(words: List[str], words_per_sent = 200, pad_str ='__PAD__') -> List[str]:
num_pad_words = words_per_sent - len(words)
pad_words = ['__PAD__'] * num_pad_words
return words + pad_words
# Add sentence padding to article
def _gen_pad_sentences(sentences: List[List[str]], words_per_sent = 200, sent_per_article = 512, pad_str ='__PAD__') -> List[List[str]]:
pad_sent = ['__PAD__'] * words_per_sent
num_pad_sentences = sent_per_article - len(sentences)
pad_sentences = [pad_sent] * num_pad_sentences
return pad_sentences
# Tokenize article sentences into words and then apply word and sentence padding functions
def _create_nn_input(article: str) -> List:
# Split article into list of sentences
sentences: List[str] = sent_tokenize(article)
# Split sentences into list of words
sentences_split_into_words = [word_tokenize(sentence) for sentence in sentences]
# Add padding to words
pad_words = [_add_word_padding(sentence) for sentence in sentences_split_into_words]
# Add padding to sentences
return pad_words + _gen_pad_sentences(pad_words)
# Group tags representing same entity together
def _group_tags(x):
# Change list of tuples into list
nested_list = [item for t in x for item in t]
# Add string 'split' before each tag containing B in 0th index of tag
res = []
for entry in nested_list:
if entry[0:2] == 'B-':
res.append('split')
res.append(entry)
nested_list[:] = res
# Split by string 'split'
grouper = groupby(res, key=lambda x: x in {'split'})
# Convert to dictionary via enumerate
conv_to_dict = dict(enumerate((list(j) for i, j in grouper if not i), 1))
# Convert dictionary of lists into list of lists
dictionary_to_list = [[k] + v for k, v in conv_to_dict.items()]
# Remove first element in each list
for x in dictionary_to_list:
del x[0]
return dictionary_to_list
# Change tag type for B if group contains two I's, then drop the I's. Otherwise, just drop the I's.
def _change_B(x):
result = x
I_tags = list(filter(lambda x: x[:2] == 'I-', x))
if len(I_tags) == 2:
second_to_last_I_tag = I_tags[-2]
new_B_tag = 'B' + second_to_last_I_tag[1:]
bb_no_Is = list(filter(lambda x: x[:2] != 'I-', x))
result = bb_no_Is.copy()
result[0] = new_B_tag
else:
bb_no_Is = list(filter(lambda x: x[:2] != 'I-', x))
result = bb_no_Is.copy()
return result
# Change B- tags with more readable label names
def _change_label(label):
if label == 'B-geo':
return 'Location'
elif label == 'B-gpe':
return 'Geopolitical Entity'
elif label == 'B-org':
return 'Company'
elif label == 'B-per':
return 'Person'
elif label == 'B-tim':
return 'Time Period'
else: return label
# Apply _change_label function to first element of each list
def _give_human_readable_label(grouping: List[str]) -> List[str]:
label = [_change_label(grouping[0])]
elements = grouping[1:]
return label + elements
# Join all but first element of each list into string
def _join_elements(x):
return [x[0]] + [' '.join(x[1:])]
# Merge lists with same first element and create dictionary
def _merge_lists(x):
d = defaultdict(list)
for i, j in x:
d[i].append(j)
return d
# Apply tag formatting functions to each entity
def _make_tags_readable(tags: List[Tuple]) -> Dict[str, List[str]]:
grouped_tags = _group_tags(tags)
# remove I-s for each element of grouped_tags
grouping_i_removed = [_change_B(grouping) for grouping in grouped_tags]
readable_groupings = [_give_human_readable_label(grouping) for grouping in grouping_i_removed]
joined_groupings = [_join_elements(grouping) for grouping in readable_groupings]
return _merge_lists(joined_groupings)
# Predict tags and apply tag formatting
def predict_tags(article):
# create list of lists
preprocessed_article = np.array(_create_nn_input(article))
predictions = model.predict(preprocessed_article)
# picks best tag for each word
predicted_label_indices = np.argmax(predictions, axis=-1)
# gets tags of each element of a
convert_tags = np.vectorize(lambda x: tags[x])
# apply convert_tags on predicted_label_indices
predicted_tags = convert_tags(predicted_label_indices)
# 2d array for labels create pairwise tuples with 2d array of words
flat_words = preprocessed_article.flatten()
flat_tags = predicted_tags.flatten()
combine_tags_words = list(zip(flat_tags, flat_words))
# function, for each tuple (lbl, word) in result, keep tuples where label (t[0]) not O
raw_tags = list(filter(lambda t: t[0] != 'O', combine_tags_words))
return _make_tags_readable(raw_tags)
# For testing
if __name__ == "__main__":
print("Setting up")
setup()
print("Setup complete")
article = """Google is a company. Bob Jones is a person."""
print("Extracting Entities")
print("Entities: ")
print(predict_tags(article))Step 6: Create a module main.py. Make sure it is stored in the same directory as clean.py and entity_recognition.py. Then download articles.csv here and store it in the same directory as main.py.
Main.py first reads in the prediction dataset as a pandas dataframe and then stores the first 10 rows as a dataframe df. Then it calls the cleaning function from clean.py. Next it calls the setup function from entity_recognition.py. Finally, it calls the predict_tags function from entity_recognition.py and applies it to each row of [‘text’] in the dataframe df.
# Import statements
# Note: If you just use 'import entity_recognition', you would have to call
# it using 'entity_recognition.predict_tags' instead of just 'predict_tags'
import clean
import pandas as pd
import entity_recognition
if __name__ == "__main__":
# Read-in prediction data
df = pd.read_csv("articles.csv", encoding="latin1")
# Set 'df' variable as first 10 rows of 'df'
df = df.head(10)
# Calls clean function in clean.py to clean 'df' dataframe
df = clean(df)
# Calls setup() function in entity_recognition.py
entity_recognition.setup()
# Applies predict_tags() function in entity_recognition.py on each row of df under ['text']
df['Entities'] = df.apply(lambda row: entity_recognition.predict_tags(row['text']), axis=1)Step 7: You can check if the code in your entity_recognition.py module works by running it on some sample text. First set the script path to entity_recognition.py in Run > Edit Configurations. Then add the test code to the bottom of entity_recognition.py. Finally click Run > Run ‘entity_recognition’.
Alternatively, you can add the test code to the bottom of entity_recognition.py and then click the green play button to the left of if __name__ == “__main__” in entity_recognition.py.
if __name__ == "__main__":
print("Setting up")
setup()
print("Setup complete")
article = """Google is a company. Bob Jones is a person."""
print("Extracting Entities")
print("Entities: ")
print(predict_tags(article))Step 8: Finally, to extract entities from each row of the target column in df and then add them to a new column called ‘Entities’: first set the script path to main.py in Run > Edit Configurations. Then, click Run > Run ‘main’.
You can instead click on the green play button to the left of if __name__ == “__main__” in main.py.