Core-term

I. Introduction

Lots of nGrams have punctuation at the begining or/and at the end. Such as:

All above n-grmas are normalized to "in details" and "in (5) details" by stripping the leading or/and ending punctuation. The normalized term is called core-term, which is the core of the term. This process is called core-term normalization.

A core term might remain internal punctuation, such as "in (5) details". Also, leading or/and ending puncutation might remian in core-term, such as "clean room(s)".

II. Algorithm

Recursively repeat the following process until term does not change or legnth = 0:

• Strip leading chars if they are punctuation except for left closed brackets, including:

  ASCII	-)}]_!@#%&*\\:;\"',.?/~+=|>$`^
  Unicode	¦§­»‐‑‒–—―’”•․‥…⁈​

• Strip ending chars if they are punctuation except for right closed brackets, including:

  ASCII	-({[_!@#%&*\\:;\"',.?/~+=|>$`^
  Unicode	¦§­«‐‑‒–—―‘“•․‥…⁈​

• strip close brackets at both ends (leading and ending position), including

  ASCII	(), [], {}, <>
  Unicode	«»‘’“”

  • Strip brackets of both lead end char if they matches and net bracket no* is = 0
  • Strip left brackets of lead char if net bracket no* is > 0
  • Strip right brackets of end char if net bracket no* is < 0
• trim

* net bracket no = total left bracket no - total right bracket no

For example,

III. Examples