Here is some of the vocabulary used by the application.
These are the basic search results in Aleph. They can either be text documents (in which case they are expected to have a number of pages and are shown as a PDF in the user interface), or tabular documents (in which case they can have multiple sheets, each with a number of records. They will be shown as tables in the user interface).
These are units documents and entity data used to group items in Aleph. This can be the name of an imported directory, database, or something more abstract, such as the name of an organisation or web site that has been ingested. It can also be used to group documents and entities relevant to a particular investigation.
These exist both for documents and collections. They are a little piece of text used to identify the origin of information, e.g. the URL of a crawled document, or the path of an imported folder. Used to avoid duplicate imports when importing the same content twice.
Represents a format and tools to store a set of documents before importing them into Aleph. It’s benefit is storing metadata alongside the actual files that are to be imported, while separating Aleph from previous workflow stages (e.g. the scraping of a web site).
Little plug-ins to the Aleph engine which import data into the system. The
included crawlers are very flexible, such as
MetaFolderCrawler, but specific crawlers can be programmed that will
import data from a specific source.
Plug-ins to the Aleph engine which accept crawled files and attempt to extract text pages or tabular rows from them so that they can be imported into the system. Ingestors for common file formats like Word documents, PDF files or CSV spreadsheets are included, but more exotic types can be supported by programming additional ingestors.
Is what Aleph runs after the documents have been ingested. They are used to extract additional information from a document. Examples include the extraction of entities and language detection. Again, new analyzers can be added through the plug-in system.
To Aleph, plugins are Python classes in a Python distutils
package which are exposed via the
entry_points mechanism. These include
crawlers, ingestors and analyzers. See
setup.py in the repository root for
This is used to describe the content of individual documents in Aleph.
Common metadata fields include:
title: a short document title (will be extrapolated from the file name, if none is given).
summary: an optional short description, usually less than 200 characters.
file_name: the basename of the imported file, e.g.
Source_Data.xlsx. If not provided, this will be guessed from the
foreign_id: see above, e.g. a source URL, or foreign systems ID
csv, without the dot
source_path: local import path, e.g.
languages: a list of lowercase two-letter ISO 3166-1 language codes, e.g
countries: a list of lowercase two-letter ISO country codes, e.g
dates: a list of ISO 8601 dates relevant to the document, eg.
keywords: a list of key phrases.
emails: email addresses extracted from the text.
headers: a hash of the headers received upon download of the document via HTTP.
content_hash: a SHA-1 checksum of the data (automatically generated).
Other fields can be added, but they will not usually be shown in the user interface.