Latent Dirichlet Allocation (LDA) is a algorithms used to discover the topics that are present in a corpus. See the slides for details.
Non-Negative Matrix Factorization is a dimension reduction technique that factors an input matrix of shape $m \times n$ into a matrix of shape $m \times k$ and another matrix of shape $n \times k$.
In text mining, one can use NMF to build topic models. Using NMF, one can factor a Term-Document Matrix of shape documents x word types into a matrix of documents x topics and another matrix of shape word types x topics. The former matrix describes the distribution of each topic in each document, and the latter describes the distribution of each word in each topic.
Non-negative Matrix Factorization (NMF) can also be used to find topics in text. The mathematical basis underpinning NMF is quite different from LDA. LDA is based on probabilistic graphical modeling while NMF relies on linear algebra. Both algorithms take as input a bag of words matrix (i.e., each document represented as a row, with each columns containing the count of words in the corpus). The aim of each algorithm is then to produce 2 smaller matrices; a document to topic matrix and a word to topic matrix that when multiplied together reproduce the bag of words matrix with the lowest error.
NMF sometimes produces more meaningful topics for smaller datasets.
NMF has been included in
scikit-learn brings API consistency which makes it almost trivial to perform Topic Modeling using both LDA and NMF.
Scikit Learn also includes
seeding options for NMF which greatly helps with algorithm convergence and offers both online and batch variants of LDA.
from sklearn.datasets import fetch_20newsgroups dataset = fetch_20newsgroups(shuffle=True, random_state=1, remove=('headers', 'footers', 'quotes')) documents = dataset.data
Downloading 20news dataset. This may take a few minutes. Downloading dataset from https://ndownloader.figshare.com/files/5975967 (14 MB)
The creation of the bag of words matrix is very easy in
scikit-learn . All the heavy lifting is done by the feature extraction functionality provided for text datasets. A
tf-idf transformer is applied to the bag of words matrix that NMF must process with the
LDA on the other hand, being a probabilistic graphical model (i.e. dealing with probabilities) only requires raw counts, so a
CountVectorizer is used. Stop words are removed and the number of terms included in the bag of words matrix is restricted to the top 1000.
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer no_features = 1000 # NMF is able to use tf-idf tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2, max_features=no_features, stop_words='english') tfidf = tfidf_vectorizer.fit_transform(documents) tfidf_feature_names = tfidf_vectorizer.get_feature_names() # LDA can only use raw term counts for LDA because it is a probabilistic graphical model tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2, max_features=no_features, stop_words='english') tf = tf_vectorizer.fit_transform(documents) tf_feature_names = tf_vectorizer.get_feature_names()
As mentioned previously the algorithms are not able to automatically determine the number of topics and this value must be set when running the algorithm. Comprehensive documentation on available parameters is available for both NMF and LDA. Initialising the W and H matrices in NMF with ‘nndsvd’ rather than random initialisation improves the time it takes for NMF to converge. LDA can also be set to run in either batch or online mode.
from sklearn.decomposition import NMF, LatentDirichletAllocation def display_topics(model, feature_names, no_top_words): for topic_idx, topic in enumerate(model.components_): print ("Topic %d:" % (topic_idx)) print (" ".join([feature_names[i] for i in topic.argsort()[:-no_top_words - 1:-1]])) no_top_words = 10 no_topics = 20 # Run NMF nmf = NMF(n_components=no_topics, random_state=1, alpha=.1, l1_ratio=.5, init='nndsvd').fit(tfidf) # Run LDA lda = LatentDirichletAllocation(n_components=no_topics, max_iter=5, learning_method='online', learning_offset=50., random_state=0).fit(tf) display_topics(nmf, tfidf_feature_names, no_top_words) display_topics(lda, tf_feature_names, no_top_words)
Topic 0: people time right did good said say make way government Topic 1: window problem using server application screen display motif manager running Topic 2: god jesus bible christ faith believe christian christians sin church Topic 3: game team year games season players play hockey win league Topic 4: new 00 sale 10 price offer shipping condition 20 15 Topic 5: thanks mail advance hi looking info help information address appreciated Topic 6: windows file files dos program version ftp ms directory running Topic 7: edu soon cs university ftp internet article email pub david Topic 8: key chip clipper encryption keys escrow government public algorithm nsa Topic 9: drive scsi drives hard disk ide floppy controller cd mac Topic 10: just ll thought tell oh little fine work wanted mean Topic 11: does know anybody mean work say doesn help exist program Topic 12: card video monitor cards drivers bus vga driver color memory Topic 13: like sounds looks look bike sound lot things really thing Topic 14: don know want let need doesn little sure sorry things Topic 15: car cars engine speed good bike driver road insurance fast Topic 16: ve got seen heard tried good recently times try couple Topic 17: use used using work available want software need image data Topic 18: think don lot try makes really pretty wasn bit david Topic 19: com list dave internet article sun hp email ibm phone Topic 0: people gun state control right guns crime states law police Topic 1: time question book years did like don space answer just Topic 2: mr line rules science stephanopoulos title current define int yes Topic 3: key chip keys clipper encryption number des algorithm use bit Topic 4: edu com cs vs w7 cx mail uk 17 send Topic 5: use does window problem way used point different case value Topic 6: windows thanks know help db does dos problem like using Topic 7: bike water effect road design media dod paper like turn Topic 8: don just like think know people good ve going say Topic 9: car new price good power used air sale offer ground Topic 10: file available program edu ftp information files use image version Topic 11: ax max b8f g9v a86 145 pl 1d9 0t 34u Topic 12: government law privacy security legal encryption court fbi technology information Topic 13: card bit memory output video color data mode monitor 16 Topic 14: drive scsi disk mac hard apple drives controller software port Topic 15: god jesus people believe christian bible say does life church Topic 16: year game team games season play hockey players league player Topic 17: 10 00 15 25 20 11 12 14 16 13 Topic 18: armenian israel armenians war people jews turkish israeli said women Topic 19: president people new said health year university school day work