tris/recoll
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Text splitter: move apos and dash character conversions to unac_except_trans.

Browse Source 
This was complicated and caused problems with highlight areas position computations in
plaintorich. Also, simplify the code for processing some dangling characters.
This commit is contained in:
Jean-Francois Dockes
2021-11-02 14:32:38 +01:00
parent a24fc7bacc
commit 728129e5ce
3 changed files with 138 additions and 192 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/sampleconf/recoll.conf
+19 -15
View File
@@ -375,29 +375,33 @@ indexstemminglanguages = english
# sub-directory.</descr></var>
#defaultcharset = iso-8859-1
# <var name="unac_except_trans" type="string"><brief>A list of characters,
# encoded in UTF-8, which should be handled specially
# when converting text to unaccented lowercase.</brief><descr>For
# example, in Swedish, the letter a with diaeresis has full alphabet
# citizenship and should not be turned into an a.
# Each element in the space-separated list has the special character as
# first element and the translation following. The handling of both the
# lowercase and upper-case versions of a character should be specified, as
# appartenance to the list will turn-off both standard accent and case
# processing. The value is global and affects both indexing and querying.
# <var name="unac_except_trans" type="string">
#
# <brief>A list of characters, encoded in UTF-8, which should be handled specially when converting
# text to unaccented lowercase.</brief>
# <descr>For example, in Swedish, the letter a with diaeresis has full alphabet citizenship and
# should not be turned into an a. Each element in the space-separated list has the special
# character as first element and the translation following. The handling of both the lowercase and
# upper-case versions of a character should be specified, as appartenance to the list will turn-off
# both standard accent and case processing. The value is global and affects both indexing and
# querying. We also convert a few confusing Unicode characters (quotes, hyphen) to their ASCII
# equivalent to avoid "invisible" search failures.
#
# Examples:
# Swedish:
# unac_except_trans = ää Ää öö Öö üü Üü ßss œoe Œoe æae Æae ffff fifi flfl åå Åå
# unac_except_trans = ää Ää öö Öö üü Üü ßss œoe Œoe æae Æae ffff fifi flfl åå Åå ’' ❜' ʼ' ‐-
# . German:
# unac_except_trans = ää Ää öö Öö üü Üü ßss œoe Œoe æae Æae ffff fifi flfl
# unac_except_trans = ää Ää öö Öö üü Üü ßss œoe Œoe æae Æae ffff fifi flfl ’' ❜' ʼ' ‐-
# . French: you probably want to decompose oe and ae and nobody would type
# a German ß
# unac_except_trans = ßss œoe Œoe æae Æae ffff fifi flfl
# unac_except_trans = ßss œoe Œoe æae Æae ffff fifi flfl ’' ❜' ʼ' ‐-
# . The default for all until someone protests follows. These decompositions
# are not performed by unac, but it is unlikely that someone would type the
# composed forms in a search.
# unac_except_trans = ßss œoe Œoe æae Æae ffff fifi flfl</descr></var>
unac_except_trans = ßss œoe Œoe æae Æae ffff fifi flfl
# unac_except_trans = ßss œoe Œoe æae Æae ffff fifi flfl ’' ❜' ʼ' ‐-
</descr></var>
unac_except_trans = ßss œoe Œoe æae Æae ffff fifi flfl ’' ❜' ʼ' ‐-
# <var name="maildefcharset" type="string"><brief>Overrides the default
# character set for email messages which don't specify