| Version at: 10/11/2018, 20:01 vs. version at: 10/11/2018, 20:02 | ||
|---|---|---|
| 1 | 1 | # How to mass import sentences |
| 2 | 2 | |
| 3 | 3 | Tatoeba has a feature to mass import sentences but it is restricted to admins. It is not very user-friendly to use and can be potentially harmful in the wrong hands. |
| 4 | 4 | |
| 5 | 5 | The page to mass import sentences is: [https://tatoeba.org/eng/sentences/import](https://tatoeba.org/eng/sentences/import) |
| 6 | 6 | |
| 7 | 7 | There are two sections on this page: |
| 8 | 8 | |
| 9 | 9 | - Single sentences |
| 10 | 10 | - Sentences and translations |
| 11 | 11 | |
| 12 | 12 | ## Single sentences |
| 13 | 13 | |
| 14 | 14 | This section allows you to import a list of sentences in a **same language**. You cannot import a list of sentences in a mix of different languages. |
| 15 | 15 | |
| 16 | 16 | The format of the file is simple: each line is one sentence. |
| 17 | 17 | |
| 18 | 18 | Example: [https://gist.github.com/trang/50497ce80f494d2be801879ed3193129](https://gist.github.com/trang/50497ce80f494d2be801879ed3193129) |
| 19 | 19 | |
| 20 | 20 | |
| 21 | 21 | The form to import sentences has 3 fields: |
| 22 | 22 | |
| 23 | 23 | - **Language of the sentences**: this is a dropdown for you to choose the language of the sentences. |
| 24 | 24 | - **File**: this is a button which will open a popup for you to select the file to import. |
| 25 | 25 | - **Numeric user id**: this is a text input for you to enter the ID of the user. |
| 26 | 26 | |
| 27 | 27 | ## Sentences and translations |
| 28 | 28 | |
| 29 | 29 | This section allows you to import sentences and their translations. Again, it is not possible to mix languages. |
| 30 | 30 | |
| 31 | 31 | The format of the file is: sentence [tab] translation. The separator between the sentence and translation must be a tab and strictly a tab. It cannot be not a sequence of several spaces. |
| 32 | 32 | |
| 33 | 33 | Example: [https://gist.github.com/trang/149ac3faddee6f3bdfde5ae97a303816](https://gist.github.com/trang/149ac3faddee6f3bdfde5ae97a303816) |
| 34 | 34 | |
| 35 | 35 | The form has 4 fields: |
| 36 | 36 | |
| 37 | 37 | - **Language of the sentences**: this is a dropdown for you to choose the language of the sentences. |
| 38 | 38 | - **Language of the translations**: this is a dropdown for you to choose the language of the translations. |
| 39 | 39 | - **File**: this is a button which will open a popup for you to select the file to import. |
| 40 | 40 | - **Numeric user id**: this is a text input for you to enter the ID of the user. |
| 41 | 41 | |
| 42 | 42 | ## How to find the user ID |
| 43 | 43 | |
| 44 | 44 | * Go to the [List of all members](https://tatoeba.org/eng/users/all). |
| 45 | 45 | * Search for the user in the sidebar. |
| 46 | 46 | * This will lead you to the latest activity page of the user. |
| 47 | 47 | * This page can also be accessed from the user profile, by clicking on the link "show latest activity" in the "Stats" section in the sidebar. |
| 48 | 48 | * Check the URL of the page. It looks like `https://tatoeba.org/eng/users/show/{number}`. The `{number}` is the user ID. |
| 49 | 49 | |
| 50 | 50 | ## Encoding requirement |
| 51 | 51 | |
| 52 | 52 | The file that you upload must be encoded in **UTF-8**. If your happen to import a file that is not encoded in UTF-8 and the sentences contain some special characters, the text imported will be messed up. |
| 53 | 53 | |
| 54 | 54 | You can [check the encoding of your file online](https://nlp.fi.muni.cz/projects/chared/). Your file is encoded in UTF-8 if the result says `Detected by Chared: utf_8`. |
| 55 | 55 | |
| 56 | 56 | There are also ways you can check offline, depending on which OS you are using. Best is to google it (for instance: [detect file encoding](http://google.com/search?q=detect+file+encoding)). |
| 57 | 57 | |
| 58 | 58 | ## Warnings |
| 59 | 59 | |
| 60 | 1. There is no safety net with this feature. Once you click "import" all the sentences will be processed and there is no easy way to cancel or revert. | |
| 60 | **(1)** There is no safety net with this feature. Once you click "import" all the sentences will be processed and there is no easy way to cancel or revert. | |
| 61 | 61 | |
| 62 | 62 | * Make sure the encoding is UTF-8. |
| 63 | 63 | * Make sure you entered the correct user ID. |
| 64 | 64 | * Make sure you selected the correct language(s). |
| 65 | 65 | * Don't import files from just anyone who asks you. Only import from trusted contributors or you could be importing a ton of bad quality sentences. |
| 66 | 66 | * Luckily you don't have to worry anymore about duplicates because Tatoeba handles that. |
| 67 | 67 | * It is recommended to make a test on the [dev website](https://dev.tatoeba.org) first, and only upload your file on the main website if it went well on the dev website. |
| 68 | 68 | |
| 69 | 2. The logs of the sentences will indicate *you* (the admin) as the person who created the sentences. If you are importing sentences for another user, make sure they are aware of it and that they don't care about it. The user will still be assigned as the owner, but the logs will state that you are contributor and the "Contributions" stats will increase on *your profile* and not on the owner's profile. | |
| 69 | **(2)** The logs of the sentences will indicate *you* (the admin) as the person who created the sentences. | |
| 70 | 70 | |
| 71 | 3. When importing sentences and translations, it is impossible to know from technically which sentence is actually the original and which one is the translation. Both sentences and translations will be considered as original. | |
| 71 | * If you are importing sentences for another user, make sure they are aware of it and that they don't care about it. | |
| 72 | * The user will still be assigned as the owner, but the logs will state that you are contributor and the "Contributions" stats will increase on *your profile* and not on the owner's profile. | |
| 73 | ||
| 74 | **(3)** When importing sentences and translations, it is impossible to know from technically which sentence is actually the original and which one is the translation. Both sentences and translations will be considered as original. | |
| 72 | 75 | |
| 73 | 76 | * The mass import feature functions as if someone was adding the sentence, then adding the translation separately as a new sentence (instead of clicking on the "Translate" button), and then using the linking feature to connect the sentence and the translation together. |
| 74 | 77 | * As a result It doesn't matter if the file you are importing has the translations in the first column and the original sentences in the second column. |
| 75 | 78 | * But as a result as well, this means that the mass import feature does not reflect accurately the reality. It loses information that is not completely critical, but could be useful in some cases (for instance for handling the permission to change the license of a sentence). |
| diff view generated by jsdifflib | ||
Version at: 10/11/2018, 20:01
# How to mass import sentences
Tatoeba has a feature to mass import sentences but it is restricted to admins. It is not very user-friendly to use and can be potentially harmful in the wrong hands.
The page to mass import sentences is: [https://tatoeba.org/eng/sentences/import](https://tatoeba.org/eng/sentences/import)
There are two sections on this page:
- Single sentences
- Sentences and translations
## Single sentences
This section allows you to import a list of sentences in a **same language**. You cannot import a list of sentences in a mix of different languages.
The format of the file is simple: each line is one sentence.
Example: [https://gist.github.com/trang/50497ce80f494d2be801879ed3193129](https://gist.github.com/trang/50497ce80f494d2be801879ed3193129)
The form to import sentences has 3 fields:
- **Language of the sentences**: this is a dropdown for you to choose the language of the sentences.
- **File**: this is a button which will open a popup for you to select the file to import.
- **Numeric user id**: this is a text input for you to enter the ID of the user.
## Sentences and translations
This section allows you to import sentences and their translations. Again, it is not possible to mix languages.
The format of the file is: sentence [tab] translation. The separator between the sentence and translation must be a tab and strictly a tab. It cannot be not a sequence of several spaces.
Example: [https://gist.github.com/trang/149ac3faddee6f3bdfde5ae97a303816](https://gist.github.com/trang/149ac3faddee6f3bdfde5ae97a303816)
The form has 4 fields:
- **Language of the sentences**: this is a dropdown for you to choose the language of the sentences.
- **Language of the translations**: this is a dropdown for you to choose the language of the translations.
- **File**: this is a button which will open a popup for you to select the file to import.
- **Numeric user id**: this is a text input for you to enter the ID of the user.
## How to find the user ID
* Go to the [List of all members](https://tatoeba.org/eng/users/all).
* Search for the user in the sidebar.
* This will lead you to the latest activity page of the user.
* This page can also be accessed from the user profile, by clicking on the link "show latest activity" in the "Stats" section in the sidebar.
* Check the URL of the page. It looks like `https://tatoeba.org/eng/users/show/{number}`. The `{number}` is the user ID.
## Encoding requirement
The file that you upload must be encoded in **UTF-8**. If your happen to import a file that is not encoded in UTF-8 and the sentences contain some special characters, the text imported will be messed up.
You can [check the encoding of your file online](https://nlp.fi.muni.cz/projects/chared/). Your file is encoded in UTF-8 if the result says `Detected by Chared: utf_8`.
There are also ways you can check offline, depending on which OS you are using. Best is to google it (for instance: [detect file encoding](http://google.com/search?q=detect+file+encoding)).
## Warnings
1. There is no safety net with this feature. Once you click "import" all the sentences will be processed and there is no easy way to cancel or revert.
* Make sure the encoding is UTF-8.
* Make sure you entered the correct user ID.
* Make sure you selected the correct language(s).
* Don't import files from just anyone who asks you. Only import from trusted contributors or you could be importing a ton of bad quality sentences.
* Luckily you don't have to worry anymore about duplicates because Tatoeba handles that.
* It is recommended to make a test on the [dev website](https://dev.tatoeba.org) first, and only upload your file on the main website if it went well on the dev website.
2. The logs of the sentences will indicate *you* (the admin) as the person who created the sentences. If you are importing sentences for another user, make sure they are aware of it and that they don't care about it. The user will still be assigned as the owner, but the logs will state that you are contributor and the "Contributions" stats will increase on *your profile* and not on the owner's profile.
3. When importing sentences and translations, it is impossible to know from technically which sentence is actually the original and which one is the translation. Both sentences and translations will be considered as original.
* The mass import feature functions as if someone was adding the sentence, then adding the translation separately as a new sentence (instead of clicking on the "Translate" button), and then using the linking feature to connect the sentence and the translation together.
* As a result It doesn't matter if the file you are importing has the translations in the first column and the original sentences in the second column.
* But as a result as well, this means that the mass import feature does not reflect accurately the reality. It loses information that is not completely critical, but could be useful in some cases (for instance for handling the permission to change the license of a sentence).
version at: 10/11/2018, 20:02
# How to mass import sentences
Tatoeba has a feature to mass import sentences but it is restricted to admins. It is not very user-friendly to use and can be potentially harmful in the wrong hands.
The page to mass import sentences is: [https://tatoeba.org/eng/sentences/import](https://tatoeba.org/eng/sentences/import)
There are two sections on this page:
- Single sentences
- Sentences and translations
## Single sentences
This section allows you to import a list of sentences in a **same language**. You cannot import a list of sentences in a mix of different languages.
The format of the file is simple: each line is one sentence.
Example: [https://gist.github.com/trang/50497ce80f494d2be801879ed3193129](https://gist.github.com/trang/50497ce80f494d2be801879ed3193129)
The form to import sentences has 3 fields:
- **Language of the sentences**: this is a dropdown for you to choose the language of the sentences.
- **File**: this is a button which will open a popup for you to select the file to import.
- **Numeric user id**: this is a text input for you to enter the ID of the user.
## Sentences and translations
This section allows you to import sentences and their translations. Again, it is not possible to mix languages.
The format of the file is: sentence [tab] translation. The separator between the sentence and translation must be a tab and strictly a tab. It cannot be not a sequence of several spaces.
Example: [https://gist.github.com/trang/149ac3faddee6f3bdfde5ae97a303816](https://gist.github.com/trang/149ac3faddee6f3bdfde5ae97a303816)
The form has 4 fields:
- **Language of the sentences**: this is a dropdown for you to choose the language of the sentences.
- **Language of the translations**: this is a dropdown for you to choose the language of the translations.
- **File**: this is a button which will open a popup for you to select the file to import.
- **Numeric user id**: this is a text input for you to enter the ID of the user.
## How to find the user ID
* Go to the [List of all members](https://tatoeba.org/eng/users/all).
* Search for the user in the sidebar.
* This will lead you to the latest activity page of the user.
* This page can also be accessed from the user profile, by clicking on the link "show latest activity" in the "Stats" section in the sidebar.
* Check the URL of the page. It looks like `https://tatoeba.org/eng/users/show/{number}`. The `{number}` is the user ID.
## Encoding requirement
The file that you upload must be encoded in **UTF-8**. If your happen to import a file that is not encoded in UTF-8 and the sentences contain some special characters, the text imported will be messed up.
You can [check the encoding of your file online](https://nlp.fi.muni.cz/projects/chared/). Your file is encoded in UTF-8 if the result says `Detected by Chared: utf_8`.
There are also ways you can check offline, depending on which OS you are using. Best is to google it (for instance: [detect file encoding](http://google.com/search?q=detect+file+encoding)).
## Warnings
**(1)** There is no safety net with this feature. Once you click "import" all the sentences will be processed and there is no easy way to cancel or revert.
* Make sure the encoding is UTF-8.
* Make sure you entered the correct user ID.
* Make sure you selected the correct language(s).
* Don't import files from just anyone who asks you. Only import from trusted contributors or you could be importing a ton of bad quality sentences.
* Luckily you don't have to worry anymore about duplicates because Tatoeba handles that.
* It is recommended to make a test on the [dev website](https://dev.tatoeba.org) first, and only upload your file on the main website if it went well on the dev website.
**(2)** The logs of the sentences will indicate *you* (the admin) as the person who created the sentences.
* If you are importing sentences for another user, make sure they are aware of it and that they don't care about it.
* The user will still be assigned as the owner, but the logs will state that you are contributor and the "Contributions" stats will increase on *your profile* and not on the owner's profile.
**(3)** When importing sentences and translations, it is impossible to know from technically which sentence is actually the original and which one is the translation. Both sentences and translations will be considered as original.
* The mass import feature functions as if someone was adding the sentence, then adding the translation separately as a new sentence (instead of clicking on the "Translate" button), and then using the linking feature to connect the sentence and the translation together.
* As a result It doesn't matter if the file you are importing has the translations in the first column and the original sentences in the second column.
* But as a result as well, this means that the mass import feature does not reflect accurately the reality. It loses information that is not completely critical, but could be useful in some cases (for instance for handling the permission to change the license of a sentence).