1- # Statistical Language Model
1+ # Statistical Language Models
22
3- ** TextAnalysis** provide following different Language Models
3+ ** TextAnalysis** provides the following different language models:
44
5- - ** MLE** - Base Ngram model.
6- - ** Lidstone** - Base Ngram model with Lidstone smoothing.
7- - ** Laplace** - Base Ngram language model with Laplace smoothing.
8- - ** WittenBellInterpolated** - Interpolated Version of witten -Bell algorithm.
9- - ** KneserNeyInterpolated** - Interpolated version of Kneser -Ney smoothing.
5+ - ** MLE** - Base n-gram model using Maximum Likelihood Estimation .
6+ - ** Lidstone** - Base n-gram model with Lidstone smoothing.
7+ - ** Laplace** - Base n-gram language model with Laplace smoothing.
8+ - ** WittenBellInterpolated** - Interpolated version of the Witten -Bell algorithm.
9+ - ** KneserNeyInterpolated** - Interpolated version of Kneser-Ney smoothing.
1010
1111## APIs
1212
13- To use the API, we first * Instantiate * desired model and then load it with train set
13+ To use the API, first instantiate the desired model and then train it with a training set:
1414
1515``` julia
1616MLE (word:: Vector{T} , unk_cutoff= 1 , unk_label= " <unk>" ) where { T <: AbstractString }
@@ -25,31 +25,31 @@ KneserNeyInterpolated(word::Vector{T}, discount:: Float64=0.1, unk_cutoff=1, unk
2525
2626(lm: :< Languagemodel> )(text, min:: Integer , max:: Integer )
2727```
28- Arguments:
28+ ** Arguments:**
2929
30- * ` word ` : Array of strings to store vocabulary.
30+ * ` word ` : Array of strings to store the vocabulary.
3131
3232 * ` unk_cutoff ` : Tokens with counts greater than or equal to the cutoff value will be considered part of the vocabulary.
3333
34- * ` unk_label ` : token for unknown labels
34+ * ` unk_label ` : Token for unknown labels.
3535
36- * ` gamma ` : smoothing argument gamma
36+ * ` gamma ` : Smoothing parameter gamma.
3737
38- * ` discount ` : discounting factor for ` KneserNeyInterpolated `
38+ * ` discount ` : Discounting factor for ` KneserNeyInterpolated ` .
3939
40- for more information see docstrings of vocabulary
40+ For more information, see the docstrings of the vocabulary functions.
4141
4242``` julia
4343julia> voc = [" my" ," name" ," is" ," salman" ," khan" ," and" ," he" ," is" ," shahrukh" ," Khan" ]
4444
4545julia> train = [" khan" ," is" ," my" ," good" , " friend" ," and" ," He" ," is" ," my" ," brother" ]
46- # voc and train are used to train vocabulary and model respectively
46+ # voc and train are used to train the vocabulary and model respectively
4747
4848julia> model = MLE (voc)
4949MLE (Vocabulary (Dict (" khan" => 1 ," name" => 1 ," <unk>" => 1 ," salman" => 1 ," is" => 2 ," Khan" => 1 ," my" => 1 ," he" => 1 ," shahrukh" => 1 ," and" => 1 … ), 1 , " <unk>" , [" my" , " name" , " is" , " salman" , " khan" , " and" , " he" , " is" , " shahrukh" , " Khan" , " <unk>" ]))
5050
5151julia> print (voc)
52- 11 - element Array {String, 1 }:
52+ 11 - element Vector {String}:
5353 " my"
5454 " name"
5555 " is"
@@ -62,42 +62,41 @@ julia> print(voc)
6262 " Khan"
6363 " <unk>"
6464
65- # you can see "<unk>" token is added to voc
66- julia> fit = model (train,2 ,2 ) # considering only bigrams
65+ # You can see the "<unk>" token is added to voc
66+ julia> fit = model (train,2 ,2 ) # considering only bigrams
6767
68- julia> unmaskedscore = score (model, fit, " is" ," <unk>" ) # score output P(word | context) without replacing context word with "<unk>"
68+ julia> unmaskedscore = score (model, fit, " is" ," <unk>" ) # score output P(word | context) without replacing context word with "<unk>"
69690.3333333333333333
7070
7171julia> masked_score = maskedscore (model,fit," is" ," alien" )
72720.3333333333333333
73- # as expected maskedscore is equivalent to unmaskedscore with context replaced with "<unk>"
73+ # As expected, maskedscore is equivalent to unmaskedscore with context replaced with "<unk>"
7474
7575```
7676
7777!!! note
7878
79- When you call `MLE(voc)` for the first time, It will update your vocabulary set as well.
79+ When you call `MLE(voc)` for the first time, it will update your vocabulary set as well.
8080
81- ## Evaluation Method
81+ ## Evaluation Methods
8282
8383### ` score `
8484
85- used to evaluate the probability of word given context (* P(word | context)* )
85+ Used to evaluate the probability of a word given its context (* P(word | context)* ):
8686
8787``` @docs
8888score
8989```
9090
91- Arguments:
91+ ** Arguments:**
9292
93- 1 . ` m ` : Instance of ` Langmodel ` struct.
94- 2 . ` temp_lm ` : output of function call of instance of ` Langmodel ` .
95- 3 . ` word ` : string of word
96- 4 . ` context ` : context of given word
93+ 1 . ` m ` : Instance of ` Langmodel ` struct.
94+ 2 . ` temp_lm ` : Output of function call of instance of ` Langmodel ` .
95+ 3 . ` word ` : String of the word.
96+ 4 . ` context ` : Context of the given word.
9797
98- - In case of ` Lidstone ` and ` Laplace ` it apply smoothing and,
99-
100- - In Interpolated language model, provide ` Kneserney ` and ` WittenBell ` smoothing
98+ - For ` Lidstone ` and ` Laplace ` models, smoothing is applied.
99+ - For interpolated language models, ` KneserNey ` and ` WittenBell ` smoothing are provided.
101100
102101### ` maskedscore `
103102``` @docs
@@ -121,28 +120,28 @@ entropy
121120perplexity
122121```
123122
124- ## Preprocessing
123+ ## Preprocessing
125124
126- For Preprocessing following functions:
125+ The following functions are available for preprocessing :
127126``` @docs
128127everygram
129128padding_ngram
130129```
131130
132131## Vocabulary
133132
134- Struct to store Language models vocabulary
133+ A struct to store language model vocabulary.
135134
136- checking membership and filters items by comparing their counts to a cutoff value
135+ It checks membership and filters items by comparing their counts to a cutoff value.
137136
138- It also Adds a special "unknown" tokens which unseen words are mapped to
137+ It also adds a special "unknown" token which unseen words are mapped to:
139138
140139``` @repl
141140using TextAnalysis
142141words = ["a", "c", "-", "d", "c", "a", "b", "r", "a", "c", "d"]
143142vocabulary = Vocabulary(words, 2)
144143
145- # lookup a sequence or words in the vocabulary
144+ # Look up a sequence of words in the vocabulary
146145
147146word = ["a", "-", "d", "c", "a"]
148147
0 commit comments