CSpell

Consumer Health Corpus

I. Introduction

A corpus relevant to consumer health data should increase the performance of CSpell. Accordingly, we established a consumer health corpus by collecting health related articles form 16 web sites that were used for answering consumer health questions:

Consumer Health Corpus (from Ashutosh's Crawler, 10.09.17)

Sources	Abbreviation	Web site Base URL	Article No.
Genetic and Rare Diseases - Diseases	gard	https://rarediseases.info.nih.gov	6484
Genetics Home Reference - Conditions	ghr	https://ghr.nlm.nih.gov/condition	1215
Genetics Home Reference - Genes	ghrgenes	https://ghr.nlm.nih.gov/gene	1439
MedlinePlus - Drugs	mplusdrugs	https://medlineplus.gov/druginfo/	1383
MedlinePlus - Medical Encyclopedia	mplusencyclopedia	https://medlineplus.gov/ency/	4425
MedlinePlus - All Health Topics	mplushealthtopics	https://www.nlm.nih.gov/medlineplus/all_healthtopics.html	1013
MedlinePlus - Herbs and Supplements	mplusherbssupplements	https://www.nlm.nih.gov/medlineplus/druginfo/herb_All.html	153/177
National Eye Institute	nei	https://nei.nih.gov/health	36
National Heart, Lung, and Blood Institute	nhlbi	http://www.nhlbi.nih.gov/health/health-topics/by-alpha	141
National Institute of Allergy and Infectious Diseases	niaid	https://www.niaid.nih.gov/diseases-conditions/all	53
National Institute of Arthritis and Musculoskeletal and Skin Diseases	niams	https://www.niams.nih.gov/health-topics/all-diseases	55
National Institute of Child Health and Human Development	nichd	https://www.nichd.nih.gov/health/topics/Pages/index.aspx	81
National Institute on Deafness and Other Communication Disorders	nidcd	https://www.nidcd.nih.gov/health/hearing-ear-infections-deafness	13/15
National Institute of Diabetes and Digestive and Kidney Disease	niddk	https://www.niddk.nih.gov/health-information	181/185
National Institute of Mental Health	nimh	http://www.nimh.nih.gov/health/topics/index.shtml	25/26
National Institute of Neurological Disorders and Stroke	ninds	https://www.ninds.nih.gov/Disorders/All-Disorders	439
Centers for Disease Control and Prevention	cdc	https://www.cdc.gov/	TBD
National Cancer Institute	cancerGov	https://www.cancer.gov/types	TBD
National Institute on Aging	nia	https://www.nia.nih.gov	TBD
National Institutes of health - Office of Research on Women's Health	womenhealth	https://orwh.od.nih.gov/	TBD

II. Algorithm

A crawler was developed to search articles that are consumer health related. The outputs are stored in XML format.
These articles are converted to text format
N-gram algorithm is applied to the text
Lower case, core-term are used to group the raw unigrams for word count
The results (WC|unigram) are used as word frequency data for CSpell

III. Consumer Health Corpus

Articles: 17,139
Sentences: 550,193
Tokens: 10,228,699
Unique Word: 192,818
Unique CoreTerm.Lc: 109,175
Dic Words in Corpus: 48690|8.5886%
Dic Words WC: 9,979,195|97.6123%

IV. Notes

The special code, such as [NUM], [EMAIL], [URL], need to be consistent with the input data for cSpell. For example, the development set used [CONTACT] for telephone number and email, which results in lower precision on context ranking. Need a cleanup on the pre-process for tagging the corpus.