tris/recoll
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

1st parallel multithreaded version of indexing which can do my home without crashing... Let's checkpoint

Browse Source
This commit is contained in:
Jean-Francois Dockes
2012-11-01 11:19:48 +01:00
parent cf96f69ee8
commit ee7d0f2ee7
23 changed files with 697 additions and 431 deletions
Download Patch File Download Diff File Expand all files Collapse all files
website/BUGS.html
+15 -11
View File
@@ -36,14 +36,8 @@
topmost section may also exist in older versions.</i></p>
<h2><a name="b_latest">recoll 1.18.002</a></h2>
<h2><a name="b_latest">recoll 1.18.003</a></h2>
<ul>
<li>It seems that a click in the snippets window can crash
recoll the very first time it is used. I could never reproduce
this on later runs and it is not known what causes the
problem. Just restart the application, and things should stay
up the next times.</li>
<li>On systems such as Debian Stable which use Evince version
2.x (not 3.x) as PDF viewer, the default "Open" command for
PDF files will not work. You need to edit the command:
@@ -77,10 +71,20 @@
<h2><a name="b_1_18_0">recoll 1.18.0</a></h2>
<ul>
<li>The missing filter recording code is broken.</li>
<li>Opening embedded documents from the Unity Lens does not
work.</li>
<li>Thumbnails are not found on newer desktops (e.g. Ubuntu
Quantal) because of a change in the freedesktop.org
"standard".</li>
<li>A bug in extracting search term from click data in the
snippet window results in passing an incorrect term to the
viewer. Only affects non-ascii terms.</li>
<li>Using the snippets window can sometimes crash the
GUI.</li>
<li>Tilde expansion is not properly performed for the
"beaglequeuedir" parameter. This only affects people who
develop scripts over the queue feature.</li>
<li>The missing filter recording code is broken.</li>
<li>Opening embedded documents from the Unity Lens does not
work.</li>
</ul>
<h2><a name="b_1_17_3">recoll 1.17.3</a></h2>
website/copydocs
+1 -1
View File
@@ -1,6 +1,6 @@
#!/bin/sh
set -x
docdir=/home/dockes/projets/fulltext/recoll/src/doc/user/
docdir=/home/dockes/projets/fulltext/17-MAINT/src/doc/user/
#docdir=/Users/dockes/projets/fulltext/recoll/src/doc/user/
#(cd $docdir;make) || exit 1
website/devel.html
+10 -95
View File
@@ -40,20 +40,13 @@
(and understand english, which can probably be assumed, you
being reading this), you can take a little time to translate
the GUI messages file.</p>
<p>The newest versions of the message files follow. There
is an empty one (the xx thing), the others are partially
<p>The newest versions of the message files follow can be found
in <a href="translations">this directory</a>. There
is an empty one (the xx one), the others are partially
translated, just needing an update for the new messages.<p>
<p>Updating the files can easily be done with
the <span class="application">Qt Linguist</span>. Contact me
for more directions if needed.</p>
<ul>
<li>Blank: <a href="translations/recoll_xx.ts">recoll_xx.ts</a></li>
<li>German: <a href="translations/recoll_de.ts">recoll_de.ts</a></li>
<li>Italian: <a href="translations/recoll_it.ts">recoll_it.ts</a></li>
<li>Russian: <a href="translations/recoll_ru.ts">recoll_ru.ts</a></li>
<li>Turkish: <a href="translations/recoll_tr.ts">recoll_tr.ts</a></li>
<li>Ukrainian: <a href="translations/recoll_uk.ts">recoll_uk.ts</a></li>
</ul>
<h1><a name="development">Development</a></h1>
@@ -67,7 +60,9 @@
tracking system</a>, these are the general areas where help or
ideas are particularly welcome:</p>
<ul>
<li>A better GUI design (both the ergonomy and the appearance).</li>
<li>A better GUI design (both the ergonomy and the
appearance). Adding missing shortcuts or fixing the menu
accelerators for exemple is easy and useful.</li>
<li>More support for the more advanced <span class=
"application">Xapian</span> concepts like relevance
@@ -95,90 +90,10 @@
<p>Reporting crashes is very useful. It can help others, and it
can get your own problem to be solved.</p>
<p>All reports are useful. But, in order to maximize usefulness,
a crash report should include a so-called stack trace, something
that indicates what the program was doing when it
crashed. Getting a useful stack trace is not very difficult,
but it may need a little work on your part (which
will then enable me do my part of the work).</p>
<p>If your distribution includes a separate package for Recoll
debugging symbols, it probably also has a page on its web site
explaining how to use them to get a stack trace. You should
follow these instructions. If there is no debugging package,
you should follow the instructions below. A little
familiarity with the command line will be necessary.</p>
<dl><dt>Compiling and installing a debugging version</dt>
<dd>
<ul>
<li>Obtain the recoll source for the version you are using
(<a
href="http://www.recoll.org/download.html">www.recoll.org</a>),
and extract the source tree.</li>
<li>Follow the instructions for
<a
href="http://www.lesbonscomptes.com/recoll/usermanual/index.html#RCL.INSTALL.BUILDING">
building Recoll from source</a> with the following
modifications:
<ul>
<li>Before running <tt>configure</tt>, edit
the <tt>mk/localdefs.in</tt> file and remove the <tt>-O2</tt>
option(s). </li>
<li>When running <tt>configure</tt>, specify the
standard installation location for your system as a prefix
(to avoid ending up with two installed versions, which
would almost certainly end in confusion). On Linux this
would typically be:
<br><tt>configure --prefix=/usr</tt>.
</li> <li>When installing, arrange for the installed
executables not to be stripped of debugging symbols by
specifying a value for the STRIP environment variable
(ie: <tt>echo</tt> or <tt>ls</tt>): <br><tt>sudo make
install STRIP=ls</tt>
</li>
</ul>
</ul></dd>
<dt>Getting a core dump</dt>
<dd>You will need to run the operation that caused the crash
inside a writable directory, and tell the system that you
accept core dumps. The commands need to be run in a shell
inside a terminal window. Ie:
<pre><tt>
cd
ulimit -c unlimited
recoll #(or recollindex or whatever you want to run).
</tt></pre>
Hopefully, you will succeed in getting the command to crash,
and you will get a core file.
</dd>
<dt>Using gdb to get a stack trace</dt>
<dd>
<ul>
<li>Install <tt>gdb</tt> if it is not already on the system.</li>
<li>Run <tt>gdb</tt> on the command that crashed and the
core file (depending on the system, the core file may be
named "core" or something else, like recollindex.core, or
core.pid), ie:
<br><tt>gdb /usr/bin/recollindex core</tt>
</li>
<li>Inside <tt>gdb</tt>, you need to use different
commands to get a stack trace for <tt>recoll</tt>
and <tt>recollindex</tt>. For <tt>recollindex</tt> you
can use the <tt>bt</tt> command. For <tt>recoll</tt>
use: <br><tt>thread&nbsp;apply&nbsp;all&nbsp;bt&nbsp;full</tt>
</li>
<li>Copy/paste the output to your report email :), and
quit <tt>gdb</tt> ("q").</li>
</ul>
</dd>
</dl>
<p>You will find help and information about producing a useful
problem report on this
<a href="https://bitbucket.org/medoc/recoll/wiki/ProblemSolvingData">
Recoll wiki page</a>.</p>
</div>
</body>
website/features.html
+2 -2
View File
@@ -317,9 +317,9 @@ application/x-tar = execm rcltar
<h2><a name="other">Other features</a></h2>
<ul>
<li>Can use <b>Beagle</b> browser plug-ins to index web
<li>Can use a Firefox extension to index visited Web pages
history. See <a href=
"http://bitbucket.org/medoc/recoll/wiki/IndexBeagleWeb">the
"http://bitbucket.org/medoc/recoll/wiki/IndexWebHistory">the
Wiki</a> for more detail.</li>
<li>Processes all email attachments, and more generally any
website/index.html.en
+17 -6
View File
@@ -84,13 +84,24 @@
<h2>News</h2>
<div class="news">
<ul>
<li>2012-10-25: the source for <a href="recoll-1.18.002.tar.gz">
recoll 1.18.002</a> is available, and this is a call to
volunteers to test it. There are binary
packages for Ubuntu and Mint Linux users, and I can build
others. See this
<li>2012-10-30: Recoll has a brand new Firefox extension for
indexing visited Web pages. This is very similar to the old
Beagle plugin, but slightly better integrated (no-fuss
installation), and with a new lease on life. See
<a href="https://sourceforge.net/projects/recollfirefox/">
Dave King's project page on sourceforge</a> and the
<a href="https://bitbucket.org/medoc/recoll/wiki/IndexWebHistory">
page about Recoll (trivial) configuration</a>.</li>
<li>2012-10-30: the source for <a href="recoll-1.18.003.tar.gz">
recoll 1.18.003</a> is available, and this is a call to
volunteers to test it. There are binary packages on the recoll
"experimental" PPA for Ubuntu and Mint Linux users, and I can
build others. See this
<a href="http://www.freelists.org/post/recoll-user/recoll-1180-prerelease">
message</a> for more information.</li>
message</a> for more information. Mini-release 003 fixes a GUI
crash and a few other issues in 002.</li>
<li>2012-10-25: a problem with a simple workaround has caused
several reported <span class="important">recollindex
website/release-1.18.html
+34 -21
View File
@@ -46,17 +46,16 @@
indexes though.</p>
<p>Case/diacritics sensitivity is off by default for this
release. It can be turned on <em>only</em> by editing
recoll.conf (see the manual). If you do so, you must reset the
index.</p>
release. It can be turned on <em>only</em> by editing recoll.conf
(see the manual). If you do so, you must then reset the index.</p>
<p>Always reset the index if installing over an older version (1.16
and older). The simplest way to do this is to quit all recoll
programs and just delete the index directory (<span
class="literal">rm&nbsp;-rf&nbsp;~/.recoll/xapiandb</span>), then
start recoll or recollindex. <span
class="literal">recollindex&nbsp;-z</span> will do the same in
most, but not all, cases.</p>
<p>Always reset the index if installing over an even older
version (1.16 and older). The simplest way to do this is to
quit all recoll programs and just delete the index directory
(<span class="literal">rm&nbsp;-rf&nbsp;~/.recoll/xapiandb</span>),
then start recoll or
recollindex. <span class="literal">recollindex&nbsp;-z</span>
will do the same in most, but not all, cases.</p>
<p>The subdirectories of xapiandb which were previously used to
store the stem expansion database (stem_english,
@@ -110,18 +109,19 @@
documents, and the <b>evince</b> viewer.</li>
<li>Recoll can now also pass a search string to the native
application.</li>
application (again, works with <b>evince</b>).</li>
<li>There is a list of mime types that should be opened with
<li>There is a list of MIME types that should be opened with
the locally configured application even when <em>Use
Desktop Preferences</em> is checked. This will permit, for
example, using evince for its page access capabilities on
PDF files, while letting the desktop handle all the other
mime types. The list is not empty by default, it contains PDF,
Postscript and DVI, so you may want to reset it after
installation if you want to keep the previous behaviour
(losing the page number functionality). This can be done
from the <em>Preferences->Gui Configuration</em> menu.</li>
example, using <b>evince</b> for its page access
capabilities on PDF files, while letting the desktop handle
all the other mime types. The list is not empty by default,
it contains PDF, Postscript and DVI, so you may want to
reset it after installation if you want to keep the previous
behaviour (losing the page number functionality). This can
be done from the <em>Preferences->Gui Configuration</em>
menu.</li>
<li>The GUI result list has a new "snippets" window for
documents with page numbers, which let the user choose a
@@ -129,7 +129,8 @@
<li>The advanced search screen now has a history
function. While the focus is in this window, you
can walk the history of searches using the up and down arrows.</li>
can walk the history of searches using the up and down
arrows.</li>
<li>We now allow multiple directory specifications in the query
language, as in: <i>dir:/home/me -dir:tmp</i></li>
@@ -149,7 +150,19 @@
<li>When running in an UTF-8 locale, and after failing to decode a
plain text file as UTF-8, indexing will try again using an 8 bit
character set heuristically chosen according to the locale
country code.</li>
language code. This uses the LANG environment variable.</li>
<li>On initial installation (when the <em>~/.recoll</em>
directory does not exist), recoll will install a list of
characters which should not be stripped of diacritics,
according to the detected national language (based on
$LANG). There are currently specific lists for German (don't
strip the umlauts), and Nordic languages (keep the letters
with circle above in addition to the German list). Other
languages currently only have exceptions which result in
decomposing ligatures (fl, fi etc.). You can have a look at
the standard recoll.conf in /usr/share/recoll/examples for
more information.</li>
<li>A new configuration variable, <tt>maxmemberkbs</tt>, has been
implemented to limit the size of archive members we process. This