doc
This commit is contained in:
Jean-Francois Dockes
parent
aba4f61e42
commit
23ec4d2e8f
@ -3356,36 +3356,71 @@ application/x-chm = execm rclchm
|
||||
<title>Filter HTML output</title>
|
||||
|
||||
<para>The output HTML could be very minimal like the following
|
||||
example:</para>
|
||||
|
||||
<programlisting><html><head>
|
||||
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
|
||||
</head>
|
||||
<body>some text content</body></html>
|
||||
example:
|
||||
<programlisting>
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
|
||||
</head>
|
||||
<body>
|
||||
Some text content
|
||||
</body>
|
||||
</html>
|
||||
</programlisting>
|
||||
</para>
|
||||
|
||||
<para>You should take care to escape some
|
||||
characters inside
|
||||
the text by transforming them into appropriate
|
||||
entities. "<literal>&</literal>" should be transformed into
|
||||
characters inside the text by transforming them into
|
||||
appropriate entities. At the very minimum,
|
||||
"<literal>&</literal>" should be transformed into
|
||||
"<literal>&amp;</literal>", "<literal><</literal>"
|
||||
should be transformed into
|
||||
"<literal>&lt;</literal>". This is not always properly
|
||||
done by translating programs which output HTML, and of
|
||||
course never by those which output plain text.</para>
|
||||
course never by those which output plain text. </para>
|
||||
|
||||
<para>When encapsulating plain text in an HTML body,
|
||||
the display of a preview may be improved by enclosing the
|
||||
text inside <literal><pre></literal> tags.</para>
|
||||
|
||||
<para>The character set needs to be specified in the
|
||||
header. It does not need to be UTF-8 (&RCL; will take care
|
||||
of translating it), but it must be accurate for good
|
||||
results.</para>
|
||||
|
||||
<para>&RCL; will also make use of other header fields if
|
||||
they are present: <literal>title</literal>,
|
||||
<literal>description</literal>,
|
||||
<literal>keywords</literal>.</para>
|
||||
<para>&RCL; will process <literal>meta</literal> tags inside
|
||||
the header as possible document fields candidates. Documents
|
||||
fields can be processed by the indexer in different ways,
|
||||
for searching or displaying inside query results. This is
|
||||
described in a <link linkend="RCL.PROGRAM.FIELDS">following
|
||||
section.</link>
|
||||
</para>
|
||||
|
||||
<para>By default, the indexer will process the standard header
|
||||
fields if they are present: <literal>title</literal>,
|
||||
<literal>meta/description</literal>,
|
||||
and <literal>meta/keywords</literal> are both indexed and stored
|
||||
for query-time display.</para>
|
||||
|
||||
<para>A predefined non-standard <literal>meta</literal> tag
|
||||
will also be processed by &RCL; without further
|
||||
configuration: if a <literal>date</literal> tag is present
|
||||
and has the right format, it will be used as the document
|
||||
date (for display and sorting), in preference to the file
|
||||
modification date. The date format should be as follows:
|
||||
<programlisting>
|
||||
<meta name="date" content="YYYY-mm-dd HH:MM:SS">
|
||||
or
|
||||
<meta name="date" content="YYYY-mm-ddTHH:MM:SS">
|
||||
</programlisting>
|
||||
Example:
|
||||
<programlisting>
|
||||
<meta name="date" content="2013-02-24 17:50:00">
|
||||
</programlisting>
|
||||
</para>
|
||||
|
||||
<para>Filters also have the possibility to "invent" field
|
||||
names. This should be output as meta tags:</para>
|
||||
names. This should also be output as meta tags:</para>
|
||||
|
||||
<programlisting>
|
||||
<meta name="somefield" content="Some textual data" />
|
||||
@ -3401,8 +3436,9 @@ application/x-chm = execm rclchm
|
||||
<meta name="somefield" markup="html" content="Some <i>textual</i> data" />
|
||||
</programlisting>
|
||||
|
||||
<para> See the following section for details about configuring
|
||||
how field data is processed by the indexer.</para>
|
||||
<para>As written above, the processing of fields is described
|
||||
in a <link linkend="RCL.PROGRAM.FIELDS">further
|
||||
section</link>.</para>
|
||||
|
||||
</sect2>
|
||||
|
||||
|
||||
@ -17,7 +17,7 @@
|
||||
# - For pages of CHM and EPUB documents where we can choose to open the
|
||||
# parent document instead of a temporary html file.
|
||||
xallexcepts = application/pdf application/postscript application/x-dvi \
|
||||
text/html|gnuinfo text/html|chm text/html|epub
|
||||
text/html|gnuinfo text/html|chm text/html|epub
|
||||
|
||||
[view]
|
||||
# Pseudo entry used if the 'use desktop' preference is set in the GUI
|
||||
|
||||
@ -46,6 +46,10 @@
|
||||
<h1>Other documentation</h1>
|
||||
|
||||
<ul>
|
||||
<li><a href="recoll_XMP/index.html">Indexing PDF
|
||||
XMP-metadata</a>: a nice exemple of customizing a Recoll
|
||||
configuration and the PDF filter to use additional
|
||||
metadata.</li>
|
||||
<li><a href="perfs.html">Index size and indexing performance
|
||||
data.</a></li>
|
||||
<li><a href="custom.html">Result list format samples.</a></li>
|
||||
|
||||
123
website/recoll_XMP/index.html
Normal file
123
website/recoll_XMP/index.html
Normal file
@ -0,0 +1,123 @@
|
||||
<html>
|
||||
<head>
|
||||
<title>Indexing PDF XMP-metadata</title>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
|
||||
<h2>Introduction</h2>
|
||||
<p>Organizing and searching a large collection of PDFs as part of a research project can be a demanding task.
|
||||
<a href="http://en.wikipedia.org/wiki/Extensible_Metadata_Platform">XMP metadata</a> stored in a PDF, such as journal title, publication year, and user-added keywords, are often useful when searching for a publication.
|
||||
Here, we describe the use of a custom Recoll filter to retrieve this metadata, an indexing configuration to store it, and result paragraph format to display it. See also a related wiki entry, <a href="https://bitbucket.org/medoc/recoll/wiki/HandleCustomField.wiki">Generating a custom field and using it to sort results</a>, for sorting results on PDF page count.
|
||||
|
||||
<h2>Saving metadata to PDFs</h2>
|
||||
<p>Bibliographic metadata can be saved in the PDF file itself. In the <a href="http://jabref.sourceforge.net">JabRef</a> bibliography manager, this is done with the "Write XMP-metadata to PDFs" menu item. Note the presence of the keywords in the screenshot below; this field is a good place to tag the PDF with any words of your choosing to describe genre, topic, etc.
|
||||
<p><img src="jabref_metadata.png">
|
||||
|
||||
<h2>Custom indexing (fields file)</h2>
|
||||
<p>Let's create two fields named "year" and "journal". The prefixes starting with "XY" are extension prefixes that are added to the terms in the Xapian database (Recoll internally does not use prefixes starting with XY). Additionally, the year and journal are stored so they can be displayed in the results list. Some other types of metadata, such as title, author and keywords, are already indexed by Recoll (the default rclpdf finds them using the <b>pdftotext</b> command) so there is no need to add those to the [prefixes] section.
|
||||
<p>Add this text to the fields file in your Recoll configuration directory (<tt>~/.recoll/fields</tt>).
|
||||
<pre>
|
||||
[prefixes]
|
||||
year = XYEAR
|
||||
journal = XYJOUR
|
||||
|
||||
[stored]
|
||||
year =
|
||||
journal =
|
||||
</pre>
|
||||
|
||||
<h2>Custom filter (rclpdf file)</h2>
|
||||
<p>This is where the heavy lifting happens. The filter should create HTML meta elements for each of the named index fields. Below is a diff between the default rclpdf and a customized one. The PDF metadata is gathered using the <b>pdfinfo</b> command. Then, <b>grep</b> and <b>sed</b> are used to extract the publication year and journal name from metadata fields beginning with "bibtex:" (part of the XMP metadata written by JabRef, in XML format). That information is fed to <b>awk</b>, which puts together the output. The crucial part in the customized awk script is the inclusion of the HTML meta elements with the names "year" and "journal".
|
||||
<p>There is some additional processing carried out by the l2html function. This replaces some LaTeX-style accents (stored in the PDF metadata if the BibTeX file contains them) with HTML entities. Only a few examples are shown here; other LaTeX accents could be processed in a similar manner. If desired, the sed commands could be modified to give UTF-8 characters instead of the HTML entities.
|
||||
<p>The l2html function also converts the LaTeX \emph{...} (emphasized text) to HTML markup for italics, <i>...</i>. With this, and the markup="html" attribute in the HTML meta elements (given in the awk script for the title and at the end of the filter for the author), italicized text and accented characters represented by HTML entities will be shown in the results.
|
||||
<p>One other thing to note: the filter changes the "Subject" HTML meta tag (created by pdftotext) to "Abstract"; this is so that the actual abstract - of the journal article, stored in the BibTeX database and written as metadata to the PDF, and reported by pdftotext as the "Subject" - is indexed independently of the title. Otherwise, terms in the "Subject" and "title" meta tags by default get indexed together by Recoll, so a title: query would actually match words appearing in the abstract.
|
||||
<p>Grab the default rclpdf for Recoll 1.18.1 (most likely <tt>/usr/share/recoll/filters/rclpdf</tt>) then apply this patch and save the result in <tt>~/.recoll/filters/rclpdf</tt> .
|
||||
<pre>
|
||||
104a105,126
|
||||
>
|
||||
> l2html()
|
||||
> {
|
||||
> # redirect the stdin so the function can be used in a pipe
|
||||
> cat |
|
||||
> # use sed to replace some accented (LaTeX format) characters
|
||||
> sed -e 's/\\"a/\&auml;/g' | # a umlaut
|
||||
> sed -e "s/\\\'a/\&aacute;/g" | # a acute
|
||||
> sed -e "s/\\\\\`a/\&agrave;/g" | # a grave
|
||||
> sed -e 's/\\u{a}/\&#x103;/g' | # a breve
|
||||
> # linebreak so multiple \emph{.*} can be replaced
|
||||
> sed -e 's/\\emph{/\n&/g' |
|
||||
> # \emph{.*} to <i>.*</i>
|
||||
> sed -e 's/\\emph{\(.*\)}/\&lt;i\&gt;\1\&lt;\/i\&gt;/g'
|
||||
> }
|
||||
>
|
||||
> # get PDF metadata
|
||||
> PDFINFO=`pdfinfo -meta "$infile" 2>/dev/null`
|
||||
> # need grep -a (--text) becuase sometimes it treats input as binary
|
||||
> YEAR=`echo "$PDFINFO" | grep -a bibtex:year | sed -e 's/<\/.*>//g' | sed -e 's/<.*>//g'`
|
||||
> JOURNAL=`echo "$PDFINFO" | grep -a bibtex:journal | sed -e 's/<\/.*>//g' | sed -e 's/<.*>//g'`
|
||||
>
|
||||
107c129
|
||||
< awk 'BEGIN'\
|
||||
---
|
||||
> awk -v year="$YEAR" -v journal="$JOURNAL" 'BEGIN'\
|
||||
111a134,136
|
||||
> yearmeta = "<meta name=\"year\" content=\""
|
||||
> journalmeta = "<meta name=\"journal\" content=\""
|
||||
> endmeta = "\">\n"
|
||||
115a141,146
|
||||
> if(doescape == 0 && $0 ~ /<\/head>/) {
|
||||
> match($0, /<\/head>/)
|
||||
> part1 = substr($0, 0, RSTART-1)
|
||||
> part2 = substr($0, RSTART, length($0))
|
||||
> $0 = part1 yearmeta year endmeta journalmeta journal endmeta part2
|
||||
> }
|
||||
133c164
|
||||
< mid = "<title>" mid "</title>"
|
||||
---
|
||||
> mid = "<meta name=\"title\" markup=\"html\" content=\"" mid "\">"
|
||||
167c198
|
||||
< '
|
||||
---
|
||||
> ' |
|
||||
168a200,203
|
||||
> # replace latex with html markup
|
||||
> l2html |
|
||||
> # add markup="html" to author meta element
|
||||
> sed -e s/\<meta\ name=\"Author\"/\<meta\ name=\"Author\"\ markup=\"html\"/g
|
||||
</pre>
|
||||
|
||||
<h2>Use the source (mimeconf file)</h2>
|
||||
<p>Recoll needs to know about your custom rclpdf. Make sure the rclpdf is executable, and add this to <tt>~/.recoll/mimeconf</tt> (replace <username> with your username).
|
||||
<pre>
|
||||
[index]
|
||||
application/pdf = exec /home/<username>/.recoll/filters/rclpdf
|
||||
</pre>
|
||||
<p>Then index away!
|
||||
<p>Note that you can also run the rclpdf script manually, e.g. <tt>rclpdf /path/to/some.pdf</tt>, to inspect the output. If things are working correctly, the <head> consists of the HTML meta elements, and the <body> contains the text of the PDF.
|
||||
|
||||
<h2>Result paragraph format</h2>
|
||||
<p>Here, the result is formatted to show the title, which is a link to open the document, in blue with underlining turned off. The next two lines contain the authors, then the journal title in green italicized text followed by year (in parentheses). The keywords are listed in red after the abstract/text snippet.
|
||||
<p>Edit this using the Recoll GUI: Preferences > GUI configuration > Result List > Edit result paragraph format string.
|
||||
<pre>
|
||||
<a href="P%N"><img src="%I" align="left"></a>
|
||||
&nbsp;<span style=font-size:1.15em><a style=text-decoration:none href="E%N">%(title)</a></span><br>
|
||||
&nbsp;%(author)<br>
|
||||
&nbsp;<font color="#009000"><i>%(journal)</i></font>&nbsp;(%(year))
|
||||
&nbsp;<table bgcolor="#e0e0e0"> <tr><td><div>%A</div></td></tr>
|
||||
</table><font color="#900000">%K</font>
|
||||
<br><br>
|
||||
</pre>
|
||||
The screenshot below also has the "Highlight color for query terms" set to <tt>black; font-weight:bold;</tt> for bold, black text (instead of the blue default). There are <a href="https://bitbucket.org/medoc/recoll/wiki/ResultsThumbnails">various methods for creating the thumbnails</a>; the ones here were made by opening the directory containing the PDFs in the Dolphin file manager (part of KDE) and selecting the Preview option.
|
||||
|
||||
<h2>A search example</h2>
|
||||
<p>The simple query is <tt>cerevisiae keyword:protein</tt>. This returns only PDFs that have the text "cerevisiae" and have been tagged with the "protein" keyword. The LaTeX-style formatting from the BibTeX database is displayed as HTML (note the italicized words in article title, and umlaut in author's name). Other queries could be made based on the PDF metadata, e.g. <tt>journal:plos</tt> or <tt>year:2013</tt> .
|
||||
<p><img src="recoll_query.png">
|
||||
|
||||
<h2>More possibilities</h2>
|
||||
<ul>
|
||||
<li>The sort buttons (up- and down-arrows) in Recoll sort the results by the modified date on the file at the time of indexing. If you want this sorting to reflect the publication year, then the timestamp should be set accordingly. If names of the PDFs contain the year (e.g. BZS2007.pdf, CKE+2011.pdf), the following one-liner would set the modified date to January 1st of the year: <tt>for i in `ls *.pdf`; do touch -d `echo $i | sed 's/[^0-9]*//g'`-01-01 $i; done</tt> . Note that the publication year could then be shown in the result list using the stored date of the file (using "%D" in the result paragraph format, and date format "%Y") instead of having to add the year to the index as shown above.
|
||||
<li>The filter can be modified to fill in the "journal" field for BibTex entries that aren't journal articles (e.g. bibtex:booktitle for "InCollection" entries).
|
||||
</ul>
|
||||
|
||||
</body>
|
||||
BIN
website/recoll_XMP/jabref_metadata.png
Normal file
BIN
website/recoll_XMP/jabref_metadata.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
BIN
website/recoll_XMP/recoll_query.png
Normal file
BIN
website/recoll_XMP/recoll_query.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 285 KiB |
Loading…
x
Reference in New Issue
Block a user