Notice

This page show a previous version of the article

Adding a New Language to the Corpus (for Developers)

Introduction

These are the instructions for adding a language to the Tatoeba corpus. Instructions for adding a language in which the Tatoeba UI will be displayed are found elsewhere.

These instructions were modified from Assembla. The sections from "In Your Local Tatoeba" onwards have not been updated yet.

FAQ for users

The FAQ for users who want to add a new language: How to request a new language.

Language icon

  1. Create the icon for the language. The icon should be a PNG file of dimension 30x20. On each icon there is (in theory) a 1px line of color #dcdcdc on the border bottom and right. Most of the icons also have gone through a luminosity change, so that they are a bit more pale than the original image. Anyway, most of it doesn't matter right now. The most important is that the icon is a PNG file of dimension 30x20.

  2. Commit the image to the repository. The icons are stored in the app/webroot/img/flags folder. Ask one of the people with repository access if you don't have it yourself and don't want to obtain it.

  3. Update the app/webroot/img/flags directory on the server, to retrieve the new images for the new languages.

Source code

There is a script that was used on the server to modify the right files to add the new language code to the appropriate files. As of 2014-03-02, an updated version of the script has been checked into our repository, though the script has not been uploaded to the server yet. The script takes the following parameters:

  • three-letter ISO 639-code (e.g., "epo" for Esperanto")
  • the English name of the language (e.g., "Esperanto")
  • the ID of a list containing at least 5 sentences in the given language (see the list of lists)
  • the string "dev" (on a development machine) or "prod" (on the server)
  • the username for the database
  • the password for the database
  • the database name

Once this script is executed, the new languages will be available on the website. The sentences that were in the list with id <list_id> will have their language set to the new language (instead of being set to language unknown). This script edits the following files:

  • app/model/sentence.php Adds the language ISO code to the $validate array. Languages that are not part of this array are not allowed.

  • app/views/helpers/languages.php Adds the language ISO code and the name to the languagesArray() method.

  • docs/generate_sphinx_conf.php Adds the language ISO code and name to the $languages array. Also adds the ISO code to the $cjkLanguages array if the language uses Chinese, Japanese or Korean characters.

In the past, we used to edit this:

  • app/controllers/components/google_language_api.php Adds the corresponding case to the google2TatoebaCode() method, if Google supports the detection for the language. See the Language enum.

but now tatodetect takes care of language detection.

After you make your changes, commit your code to the repository, or have someone do it for you. See Repositories.

In your local Tatoeba

  • Connect to mysql and select the database.
  • If you haven't done it yet, run the following script: docs/database/scripts/add_new_language.sql. It will create a procedure to easily add a new language and do the necessary updates in the database.
  • CALL add_new_language(iso_code, list_id, tag_name);
  • Read the comments in add_new_language.sql to have examples of the procedure.
  • Test that the language detection works (or can work) by adding a sentence with 'auto-detect'. There should be on Tatoeba a list of sentences in the language in question (named after the language in question).
  • Test that you can change the language of a sentence into the language in question.
  • Check that the count displays properly in the languages stats.
  • If it's all fine, commit and refer to the ticket #225 in your comment (=> re #225) and indicate the languages that were added. Also refer to any separate tickets that were added to track adding these languages in particular. The syntax for referring to multiple tickets is described here.

On the dev

  • Go to the 'dev' repository.
  • Update it, if necessary.
  • Connect to the mysql database of the dev version.
  • CALL add_new_language(iso_code, list_id, tag_name);
  • Test the same things you have tested in local.

On the prod

  • If everything is fine with the dev, go to the 'prod' repository.
  • htop
  • Check that the load is below 2.
  • Update the repository if necessary
  • Connect to the mysql database of the prod version.
  • CALL add_new_language(iso_code, list_id, tag_name);
  • exit
  • Check that the sentences that were in the list and tags have now the appropriate icon.
  • Check that the language appears in the languages stats.
  • cp /usr/local/etc/sphinx.conf /usr/local/etc/sphinx.conf.old
  • php generate_sphinx_conf.php > /usr/local/etc/sphinx.conf
  • Change the necessary things in the new config file (user, password, database and port). Look at the old conf file for reference.
  • indexer --all --rotate & disown

Historical information only

We used to follow this procedure for adding a new language (example: French):

  • Create folder : /app/locale/fre/LC_MESSAGES
  • Copy-paste default.pot into this folder
  • Change it into default.po
  • Open default.po with PoEdit (http://www.poedit.net/) and translate.
  • Save. It will generate a *.mo file, which is used when replacing strings at runtime.

and when new strings were added:

  • Follow the cake i18n instructions to generate the up-to-date POT file.
  • Open the PO file (PO, not POT).
  • In the menu : Catalog > Update from POT file…
  • Choose the POT file that was newly generated

The language of the page is set through the URL. Example: http://localhost/tatoeba2/fre/sentences/index

Resources:

  • http://blog.jaysalvat.com/articles/choix-des-langues-par-url-dans-cakephp.php
  • http://www.formation-cakephp.com/41/multilingue-18n-l10n

Actions