Skip to content

README.adoc

Latest commit

 

History

History
191 lines (139 loc) · 6.84 KB

README.adoc

File metadata and controls

191 lines (139 loc) · 6.84 KB
Raw

MT LibreTranslate Component

Table of Contents

Intention

MT tool that uses LibreTranslate to translate questions into English.

Behavior

As a Qanary component, it reads the question and the annotations produced by upstream components from the Qanary triplestore (the shared process memory), performs its task — translate the question into another natural language — and writes the result back as standard Qanary (Web Annotation / qa:) annotations. See Input specification / Output specification below for the exact annotation contract.

Characteristics

  • Task type: Machine Translation (MT).

  • Runtime: containerised; some models/services may need to be downloaded on first run.

Implementation

  • Language: Python 3.

  • Major frameworks: FastAPI, Uvicorn, qanary-helpers (with SentencePiece, SPARQLWrapper, torch, transformers).

  • Tests: 2 pytest file(s) under tests/ (test_lang_utils.py, test_mt_libretranslate.py).

Build and run

Build and run the containerised component (configure the .env / environment first):

docker compose up --build
# or, for local development:
pip install -r requirements.txt && python run.py

Input specification

Not applicable as the textual question is a default parameter.

Output specification

@prefix qa: <http://www.wdaqua.eu/qa#> .
@prefix oa: <http://www.w3.org/ns/openannotation/core/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<urn:qanary:output1> a qa:AnnotationOfQuestionTranslation ;
    oa:hasTarget <urn:myQanaryQuestion>;
    oa:hasBody "translation_result"@en ;
    oa:annotatedBy <urn:qanary:PythonMTLibreTranslate> ;
    oa:annotatedAt "2001-10-26T21:32:52"^^xsd:dateTime .

Usage

  1. Clone the Git repository of the collected Qanary components:

git clone https://github.com/WDAqua/Qanary-question-answering-components.git

  1. Switch to the component’s directory:

cd Qanary-question-answering-components/qanary-component-Python-MT-LibreTranslate

  1. Set the environment variables in the .env file

SERVER_PORT=40120
SPRING_BOOT_ADMIN_URL=http://qanary-pipeline-host:40111
SERVER_HOST=http://public-component-host
SPRING_BOOT_ADMIN_USERNAME=admin
SPRING_BOOT_ADMIN_PASSWORD=admin
SERVICE_NAME_COMPONENT=LibreTranslate
TRANSLATE_ENDPOINT=http://localhost:5000/translate
LANGUAGES_ENDPOINT=http://localhost:5000/languages
SOURCE_LANGUAGE=de
TARGET_LANGUAGE=en

The parameters description:

  • SPRING_BOOT_ADMIN_URL — URL of the Qanary pipeline (see Step 1 and Step 2 of the tutorial)

  • SPRING_BOOT_ADMIN_USERNAME — the admin username of the Qanary pipeline

  • SPRING_BOOT_ADMIN_PASSWORD — the admin password of the Qanary pipeline

  • SERVER_HOST — the host of your Qanary component without protocol prefix (e.g., http://). It has to be visible to the Qanary pipeline (i.e., a callback from the Qanary pipeline can be executed).

  • SERVER_PORT — the port of your Qanary component (has to be visible to the Qanary pipeline)

  • SERVICE_NAME_COMPONENT — the name of your Qanary component (for better identification)

  • TRANSLATE_ENDPOINT — the LibreTranslate endpoint to be used for translation

  • LANGUAGES_ENDPOINT — the LibreTranslate endpoint returning a list of supported languages

  • SOURCE_LANGUAGE — (optional) the default source language of the translation

  • TARGET_LANGUAGE — (optional) the default target language of the translation

    1. pull the LibreTranslate image:

docker-compose pull libretranslate

Note: Downloading the required language models might take several minutes during which the service will be unavailable. Use volumes to cache downloaded models.

Note: If you have access to the LibreTranslate API (though an API key), you may change the TRANSLATE_ENDPOINT to the official URL. In this case, the libretranslate service from the docker-compose file is not needed.

  1. Build the Docker image:

docker-compose build

  1. Run the latest version with docker-compose:

docker-compose up latest

After successful execution, component creates Qanary annotation in the Qanary triplestore:

GRAPH <uuid> {
  ?a a qa:AnnotationOfQuestionTranslation .
  ?a  oa:hasTarget <urn:myQanaryQuestion> .
  ?a  oa:hasBody "translation_result"@ISO_639-1 language code
  ?a  oa:annotatedBy <urn:qanary:app_name> .
  ?a  oa:annotatedAt ?time .
}

Support for multiple Source and Target Languages

This component relies on the presence of one of more existing annotations that associate a question text with a language. This can be in the form of an AnnotationOfQuestionLanguage, as created by LD components, or an AnnotationOfQuestionTranslation as created by MT components.

It supports multiple combinations of source and target languages. You can specify a desired source and target language independently, or simply use all available language pairings.

If a SOURCE_LANGUAGE is set, then only texts with this specific language are considered for translation. If none is set, then all configured source languages will be used to find candidates for translation.

Similarily, if a TARGET_LANGUAGE is set, then texts are only translated into that language. If none is set, then the texts are translated into all target languages that are supported for their respective source language.

Note that while configured source languages naturally determine the possible target languages, the configured target languages also determine which source languages can be supported!

Pre-configured Docker Images

You may use the included file docker-compose-pairs.yml to build a list of images that are preconfigured for specific language pairs. Note that if you intend to use these containers at the same time, you need to assign different SERVER_PORT values for each image.

docker-compose -f docker-compose-pairs.yml build

Endpoints

  • /about — (GET) a short service description

  • /health — (GET) the status of the connection to a LibreTranslate service

  • /languages — (GET) a list of supported source languages with "en" as target language

  • /annotatequestion — (POST) standard endpoint for Qanary components

How To Test This Component

This component uses the pytest. The necessary environment variables have to be configured in pytest.ini.

Note: The use of a virtual environment is encouraged for this.

First, install the requirements with pip install -r requirements.txt.

Then run the local tests with the command pytest.