Performance Tests on Orthographic Similarity Score

I. Test Setup

• Data: Training Set
• Gold Standard: non-word only
• Dictionary: CSpell (Lexicon-based)
• Corpus: none
• Ranking: Orthographic ranking

II. Test Results

• Tests on token (edit distance), phonetic, and overlap similarity scores:

  ID	Ranking	Precision	Recall	F1
  0-1	Edit Distance	0.7606	0.7636	0.7621
  0-2	Phonetic	0.7490	0.7519	0.7505
  0-3	Overlap	0.7542	0.7571	0.7556

• Tests on orthographic similarity scores using various weighting factors (WF) of token (edit distance), phonetic, and overlap similarity scores:

  ID	Edit Distance	Phonetic	Overlap	Precision	Recall	F1	Notes
  1	1.00	1.00	1.00	0.7580	0.7610	0.7595	same ratio of WF
  2	0.95	0.95	0.95	0.7580	0.7610	0.7595
  3	0.90	0.90	0.90	0.7580	0.7610	0.7595
  4	1.00	0.90	0.90	0.7593	0.7623	0.7608	Increase 1 WF
  5	0.90	1.00	0.90	0.7580	0.7610	0.7595
  6	0.90	0.90	1.00	0.7580	0.7610	0.7595
  7	0.80	0.90	0.90	0.7580	0.7610	0.7595	Decrease 1 WF
  8	0.90	0.80	0.90	0.7593	0.7623	0.7608
  9	0.90	0.90	0.80	0.7580	0.7610	0.7595
  10	1.00	0.80	0.90	0.7593	0.7623	0.7608	Try and error by increasing Edit distance, decreasing phonetic
  11	1.00	0.80	0.85	0.7593	0.7623	0.7608
  12	1.00	0.70	0.80	0.7606	0.7636	0.7621
  13	1.00	0.70	0.90	0.7593	0.7623	0.7608
  14	1.00	0.00	0.00	0.7606	0.7636	0.7621
  15	1.00	0.50	0.80	0.7606	0.7636	0.7621
  16	1.00	0.60	0.80	0.7606	0.7636	0.7621
  17	1.00	0.65	0.80	0.7606	0.7636	0.7621
  18	1.00	0.65	0.90	0.7606	0.7636	0.7621
  19	1.00	0.65	0.90	0.7606	0.7636	0.7621
  20	1.00	0.75	0.90	0.7593	0.7623	0.7608
  21	1.00	0.85	0.90	0.7593	0.7623	0.7608

  III. Discussion

  • From the test 0-1-0-3, the order of better ranking in orthographic is Edit-distance, overlap, phonetic
  • The result of tests 1-3 are the same. That is the same ratio of weighting factors leads to same results
  • From the results of test 4-6, we observed the higher the weighting factor of edit distance similarity score, the better the F1 score.
  • From the results of test 7-9, we observed the lower the weighting factor of phonetic similarity score, the better the F1 score.
  • Find the best F1 by try and error on tests 10-21