N-gram Set by Prediction Filter

A new approach of prediction filter is developed to resolve issues of limited memory. This approache retrieve an approximate n-Gram as an alternative approach. However, this approach is not comprehensive and should be replace by a more thorough approach.

I. Prediction Filter:

Use the frequency (NWC) of normalized (n)-gram terms as filter to generate (n+1)-gram terms:

• Only high frequency n-gram terms are used to generate multiwords for our applications (to add in to Lexicon). Low frequency n-gram terms are usually error prone multiwords (e.g. typos).
• Among these n-gram terms, most of them has very low frequency (as the long tail of dinosaur shown on the single words frequency spectrum)
• The frequency (WC) of normalized n-gram terms is used to reflect the real usage.
• If a n-gram term has low NWC, the associated (n+1)-gram terms has lower (or equals) NWC. For example, unigram term "zytochemical" has NWC of 2. The associated bigram of "zytochemical" is "zytochemical findings" is also 2.
• Let says the n-gram term is "n-gram term", the associated (n+1)-gram terms must be:
  • "prefix n-gram term"
  • "n-gram term suffix"
• Use the frequency (NWC) of normalized (n)-gram terms as filter to generate (n+1)-gram terms to generate high frequency (n+1) gram terms that fit into to our computing resources (60Gb, which stores about 230 million of n-gram terms)a In other words, this approach retrieve the top NWC (frequency) 230 million of n-gram terms (for n = 1 ~ 5, respectively).

II. N-gram Set with Prediction Filter:

• Get the nGram (n=1, uniGram) from section I.
• Run the normalization on nGram and retrieve uniGram with normalized terms|word count
• Manually review results and select a threshold (wc >= 50)
• Retrieve a table with all original terms with normalized terms above the selected threshold
• Generate biGram (n+1) as follows:
  • Filter out the biGram ((n+1)-gram) term if either the starting or ending words in biGram terms are not in the unigram (n-gram) threshold table (below the threshold, because the frequency is too low). These low frequency term will be filtered out after they are generated (even we don't filter them out at this stage anyway).

• Repeats the above processes for n = 2 ~ 5

III. Example Walk-through (MEDLINE.2014):