Version at: 02/03/2014, 17:01 vs. version at: 02/03/2014, 18:38
11GSoC ideas for student projects
22===============================
33
44This page lists example ideas for students who would like to take part in [Google Summer of Code](http://www.google-melange.com/) and be mentored by Tatoeba. To quote [GSoC FAQ](http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2013/help_page#3._What_is_an_Ideas_list):
55
66<blockquote>
77<p>An Ideas list should be a list of suggested student projects. This list is meant to introduce contributors to your project's needs and to provide inspiration to would-be student applicants. It is useful to classify each idea as specifically as possible, e.g. "must know Python" or "easier project; good for a student with more limited experience with C++." If your organization plans to provide a proposal template for the students, it would be good to include it on your Ideas list.</p>
88
99<p>Keep in mind that your ideas list should be a starting point for student proposals; we've heard from past mentoring organization participants that some of their best student projects are those that greatly expanded on a proposed idea or were blue-sky proposals not mentioned on the ideas list at all. A link to a bug tracker for your open source organization is NOT an ideas list.</p>
1010
1111<p>You can check out the <a href="http://community.kde.org/GSoC/2011/Ideas">Ideas list for KDE</a> for Google Summer of Code in 2011 to get an idea of what we’re looking for in an ideas list. </p>
1212</blockquote>
1313
1414If you're a student, you're invited to discuss any of these ideas, as well as propose your own. Feel free to [get familiar with the codebase](main#developers) already. To contact us, use one of these:
1515
1616* [Tatoeba Wall page](http://tatoeba.org/wall/index)
17* Email/Google groups: [Tatoeba GSoC mailing list](https://groups.google.com/forum/?fromgroups=#!forum/tatoeba-gsoc)
17* Email/Google groups: [Tatoeba's dev mailing list](https://groups.google.com/forum/#!forum/tatoebaproject)
1818* IRC: [Tatoeba on #Freenode](irc://irc.freenode.net/tatoeba), [Webchat](http://webchat.freenode.net?channels=tatoeba)
1919* XMPP: [Tatoeba conference room on chat.tatoeba.org](xmpp:tatoeba@chat.tatoeba.org?join)
2020
2121
2222Current site
2323------------
2424
2525Extending current PHP site:
2626
2727* programming in PHP with the CakePHP framework
2828* shell tools for maintenance (e.g.: better export scripts)
2929* JavaScript
3030
3131### Better export scripts
3232
3333Currently CSV dumps are done weekly. They require the database to be switched into a read-only mode, take 5~10 minutes, and do not contain some important information, such as tag creator and comments. CSV dumps are important for people who cooperate with Tatoeba by creating additional tools, so their quality is vital for healthy collaboration.
3434
3535**Deliverables**: An database export mechanism that:
3636
3737 * Dumps all interesting information (everything currently in the data dumps plus modification history, sentence comments, the wall, etc.).
3838 * Can create incremental dumps, making them faster, which will allow them to be made more frequently.
3939 * Provides an interface for collaborators to get notifications about new dumps and allows automatic access.
4040 * (Advanced) Provides a stream of updates via web sockets or a similar mechanism.
4141
4242**Prerequisite knowledge**: a scripting language (Python preferred), PHP, MySQL.
4343
4444### Administrative Scripts
4545
4646The current site has experienced a couple of crashes and instabilities. We would like to increase the number of users who are fully capable of administering it with ease and reliability. In order to ease administration and quickly recover from disaster, a number of scripts covering common administrative tasks are needed.
4747
4848**Deliverables**:
4949
5050*Shell scripts* that cover:
5151
5252 * backup
5353 * restoring of backups
5454 * export of data
5555 * import of data
5656 * adding new languages
5757 * deduplication of sentences
5858 * indexing of sentences by the search daemon
5959 * preserving existing translations when typos in the source UI strings are fixed
6060 * getting external services up and running
6161 * updating the production site from the repository
6262 * deployment on a real server from scratch
6363 * deployment on a development machine from scratch
6464 * monitoring the server and logging load and activity
6565 * other necessary tasks
6666
6767An *administrative interface* accessible by admins to manually execute any of these tasks on the server with progress bars and statistics.
6868
6969**Prerequisite knowledge**: a scripting language (bash, Python, Perl, etc..), possibly familiarity with a build system (ansible, vagrant), and possibly familiarity with setting up and maintaining a monitoring system (newrelic, nagios, cacti, munin)
7070
7171### Testing Suite
7272
7373The current website doesn't have any tangible automated tests or any form of continuous integration. Any new code that gets added to the repository can break the existing website and most testing is done manually at this point. In order to make the code future proof and give the users a stable experience, a battery of tests for the main functionality of the website is needed.
7474
7575**Deliverables**: A number of tests that create a test database and emulate a browser, testing some or all of the functionality that the website currently offers. Refer to this [list](http://en.wiki.tatoeba.org/articles/show/functionality-test-list)
7676
7777**Prerequisite knowledge**: a scripting language (Python, PHP, etc...), MySQL, and a web testing framework (selenium, or something similar).
7878
7979### API
8080
8181The Tatoeba database is used either through the main website interface or through data dumps. Having a real API that can be called through AJAX and return machine-readable results would provide real-time access for external applications.
8282
8383**Deliverables**: A web application that provides a set of API calls for data stored in the current database. The API should cover all data available through the current web interface, including sentence comments, wall comments, recently-added sentences and top recent contributors.
8484
8585**Prerequisite knowledge**: a web application language (Python or PHP preferred), MySQL, and a data exchange format such as JSON or XML.
8686
8787### API Compliant Interface
8888
8989With the presence of an API and an API [spec](https://github.com/trang/tatoeba-api/wiki/Tatoeba-API-specification-2) most direct SQL queries in the current codebase will become obsolete and the preferred way to do it would be through the API. So a rewrite or a completely new Interface will be the next logical step.
9090
9191**Deliverables**: A rewrite of the current codebase to use the API instead of SQL queries or a completely new interface in another language, preferably in a javascript framework.
9292
9393**Prerequisite knowledge**: strong knowledge of CakePHP, familiarity with the API, and familiarity with the current codebase, or strong knowledge of a web framework (django for example) or a javascript framework (angularjs for example).
9494
9595### Improvements in user interface for end users
9696
9797Tatoeba now handles several kinds of queries, but more are desired. The translation interface, in particular, needs improvement. Examples of desired types of queries:
9898
9999* Get all sentences in a given language by a given user that have not been translated into a given language. For example: *Show me all English sentences by user "CK" not yet translated into Japanese.*
100100
101101* Same as above, but limited to sentences with audio. For example: *Show me all English sentences by "CK" with audio that have not been translated into Japanese.*
102102
103103* Get all sentences by native speakers of a given language not yet translated into a given language. For example: *Show me all Finnish sentences by native speakers not translated into Hungarian.*
104104
105105* Get all sentences in a given language with a certain tag not translated into a given language. For example: *Show me all Georgian sentences with the tag "restaurant" not translated into Armenian.*
106106
107107* Same as above, but limited to sentences by native speakers not translated into a given language. For example: *Show me all Korean sentences by native speakers with the tag "weather" not translated into Japanese.*
108108
109109* Get all sentences in a given language under a certain length not yet translated into a given language. For example: *Show me all Japanese sentences fewer than 50 characters in length not translated into French.*
110110
111111* Same as above, but limited to native speaker sentences.
112112
113113* Same as above, but limited to sentences by a given user.
114114
115115* Get all sentences by native speakers of a given language that match a given search keyword that have not been translated into a given language. For example: *Show all English sentences with the word "mountain" not translated into Japanese.*
116116
117117* Same as above, but limited to native speaker sentences.
118118
119119* Same as above, but limited to sentences by a given user.
120120
121121**Deliverables:** Implementation of some or all of the above. Project might include additional queries. It would be highly desired to provide a generic way of adding new types of queries.
122122
123123**Prerequisite knowledge**: PHP, CakePHP.
124124
125125### Allow users to follow each other
126126
127127This feature would allow users to keep track of newly created sentences of other users, just like Twitter does. One could be able to get notified of new sentences of users he or she's following, and to browse them. Public and private visibility of who’s following who should be discussed prior to development.
128128
129129**Deliverables:** a mean to follow one or more users ; a page that displays the sentences of the followed users and allows to browse and search through ; configurable notifications about new sentences of the followed users ; displaying of who’s following who.
130130
131131**Prerequisite knowledge**: PHP, CakePHP.
132132
133133### Word requests
134134
135135This feature would allow users to request example sentences that show the correct usage of a given word or phrase. Contributors could browse lists of 'requested words' and add sentences that include them.
136136
137137People interested in this idea should consider and discuss possible implementations details prior to development. Typical questions include:
138138
139139* What should be the scope of the lists (per-language, per-user…)?
140140* How should the lists be maintained?
141141* Can we indicate that a 'requested word' now has enough example sentences? If yes, how?
142142* What’s the lifecycle of a typical requested word?
143143* What if users want to express additional information in their requests, such as the context or sense for the requested word?
144144* What about synonyms, inflections… ?
145145
146146**Deliverables:** a mean to express the need of example sentences of a given word ; a way to easily contribute new sentences that shows example of wanted words
147147
148148**Prerequisite knowledge**: PHP, CakePHP.
149149
150150### Show pronunciation in IPA for sentences
151151
152152IPA stands for "International Phonetic Alphabet" and is used to describe pronunciation of human languages in an unambiguous way. As such, it helps learning languages whose pronunciation rules are complex (e.g., English). Tatoeba could display IPA pronunciation for each sentence in basically the same way it currently displays pronunciation for Japanese using kana. One possible way of performing the task is to use an external library or application to prepare IPA annotations. For example, [eSpeak](http://espeak.sourceforge.net/) seems to be able to handle several popular languages and has an IPA converter.
153153
154154**Deliverables:** A mechanism that shows IPA pronunciation for some languages (chosen by the student). This can be done server-side (as a standalone service or part of existing code) or client-side (using JavaScript). Mechanism should allow pre-generating pronunciation descriptions and should provide means to manually edit pronunciation later. Mechanism can rely on 3rd party tool to generate pronunciation descriptions.
155155
156156**Prerequisite knowledge**: web technology, some web application stack (PHP, Python or CppCMS preferred).
157157
158158New site & CppCMS
159159-----------------
160160
161161Helping Sysko with tatowiki, tatodb. Extending [CppCMS](http://cppcms.com/wikipp/en/page/main). As the new site is still mostly being planned, there are no specific project ideas for the moment. Please ask on the IRC channel for more information. Note that many projects in this category will have an experimental nature, and their scope highly depends on your skills.
162162
163163Standalone user tools
164164---------------------
165165
166166Work on tools such as the following:
167167
168168* [shtooka recorder](http://a4esl.com/temporary/tatoeba/shtooka/) or [swac-record](http://zmoo.fr/swac-tools/) for recording audio for sentences
169169* [tatoparser](https://github.com/qdii/tatoeba_parser) for retrieving sentences that match regular expressions
170170* [katoeba](https://github.com/sadhen/katoeba)
171171
172172Create new tools, such as apps for smartphones, for ordinary or advanced contributors.
173173
174174### Android/iPhone application
175175
176176iPhone users are about 12%, and Android users about 7% of the site visitors. It might help them immensely if they could use a dedicated application.
177177
178178**Deliverables**: A smartphone application for easy access to Tatoeba. Examples of features:
179179
180180* Querying the online Tatoeba site
181181* Adding sentences
182182* Translating
183183* Performing typical corpus maintenance tasks (linking/unlinking sentences, changing sentence language, tagging, etc.)
184184* Access to wall and sentence comments
185185* Recording voice
186186* Offline database access (more difficult!)
187187
188188Note: It is not expected to implement all of these features during a single GSoC event. Depending on your skills, you might prepare a proposal for a basic set of features (if you don't have much experience in mobile development yet) or a more complex or targeted application (if you do have experience and want to prepare something more feature-complete).
189189
190190**Prerequisite knowledge**: Java and Android development or iPhone and iOS developement; using web services.
191191
192192### Streamlined linking of multiple sentences
193193
194194Where multiple sentences in a source language have the same translation in the target language, make it easy to link those source sentences to the same target translation. Collecting the sentences that are likely to have the same translation could be as simple as presenting sentences in order of creation, since variants of a sentence that vary only in, e.g., the number of the pronoun (where the singular and plural forms of the second person map to the same word in English) are likely to be entered consecutively.
195195
196196**Prerequisite knowledge**: JavaScript, possibly Java, possibly SQL
197197
198198### Help bots
199199
200200Produce bots like [those in Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Bots), to help with maintenance and repetitive tasks such as fixing common mistakes, wrong flag, etc. As on Wikipedia, users that are actually bots should be identified somehow on the website side. Ideally, create a library that could be used as a base to create bots that interact with the Tatoeba website.
201201
202202Note: this idea would highly benefit from having a real API, which is another project listed here.
203203
204204**Prerequisite knowledge**: web
205205
206206Other ideas
207207-----------
208208
209209External services, like [CK's Temporary Tatoeba site](http://a4esl.com/temporary/tatoeba/). Other ideas.
210210
211211### XMPP Integration for Tatoeba
212212
213213Use the XMPP communications protocol to integrate the manipulation of sentences, comments, and wall posts, as well as live feeds of latest comments, sentence additions, and wall posts, with services such as pubsub.
214214
215215**Deliverables**:
216216
217217* An XEP that outlines the protocol tatoeba would use for all of those operations over XMPP ready to be submitted to the XSF
218218* An implementation of this XEP in XMPP clients as plugins, poezio and gajim are top priorities.
219219* An implementation of this XEP server side as a module, prosody is a top priority.
220220
221221**Prerequisite Knowledge**: XMPP, PubSub, Python, Lua, Familiarity with prosody/gajim/poezio codebases and plugin architecture
222222
223223
224224### SRS deck generator
225225
226226Spaced Repetition Systems such as Anki and Mnemosyne are popular tools for learning languages. However, preparing a good SRS deck is a time-consuming task. Therefore, an automated way to generate a deck from a list of sentences (e.g., sentences on a Tatoeba list, sentences tagged by some specific tag, etc.) would help language learners.
227227
228228**Deliverables**: an application (preferably a web-based one) that would use Tatoeba database (for example in the form of a weekly CSV data dump) to create SRS decks for major flash card applications. Examples of features:
229229
230230* Generate a simple deck from a Tatoeba list, tag, or search query.
231231* Generate an N+1-style deck based on user's list of known words and Tatoeba database. (User gives a list of N words that s/he already knows. System chooses a new sentence where exactly one word is unknown, and the rest belong to the already known set.)
232232* Generated decks have proper internal structure (as for Anki decks: proper field scheme is used to store knowledge, so editing is easy).
233233
234234**Prerequisite knowledge**: any web stack, however Django or CppCMS are prefered; Python or C++; knowledge about SRS.
235235
236236### Browsable graph of sentence links
237237
238238Given a sentence, display it as a [graph](https://en.wikipedia.org/wiki/Graph_%28data_structure%29) [like this](http://blog.tatoeba.org/2010/02/how-to-be-good-contributor-in-tatoeba.html#rule2) the linked sentences up to a given depth. The main purpose of such a graph is to show users how Tatoeba is structured at a glance. The current interface doesn’t provide such a view, but it’s important that users understand the actual structure of Tatoeba. This idea could be freely extended to a complete interface allowing linking and unlinking with a click, filter by language, edit sentences, or whatever you can think of.
239239
240240**Deliverables**: a web application or a client-side JavaScript program that provides a graph view of a group of sentences, and allows manipulating them. Code can either operate on database directly or use existing or planned APIs.
241241
242242Note that this idea can be implemented as part of the current code base in PHP or as an experimental service for a new CppCMS site (preferred).
243243
244244**Prerequisite knowledge**: PHP, Python or CppCMS.
diff view generated by jsdifflib

Version at: 02/03/2014, 17:01

GSoC ideas for student projects
===============================

This page lists example ideas for students who would like to take part in [Google Summer of Code](http://www.google-melange.com/) and be mentored by Tatoeba. To quote [GSoC FAQ](http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2013/help_page#3._What_is_an_Ideas_list):

<blockquote>
<p>An Ideas list should be a list of suggested student projects. This list is meant to introduce contributors to your project's needs and to provide inspiration to would-be student applicants. It is useful to classify each idea as specifically as possible, e.g. "must know Python" or "easier project; good for a student with more limited experience with C++." If your organization plans to provide a proposal template for the students, it would be good to include it on your Ideas list.</p>

<p>Keep in mind that your ideas list should be a starting point for student proposals; we've heard from past mentoring organization participants that some of their best student projects are those that greatly expanded on a proposed idea or were blue-sky proposals not mentioned on the ideas list at all. A link to a bug tracker for your open source organization is NOT an ideas list.</p>

<p>You can check out the <a href="http://community.kde.org/GSoC/2011/Ideas">Ideas list for KDE</a> for Google Summer of Code in 2011 to get an idea of what we’re looking for in an ideas list. </p>
</blockquote>

If you're a student, you're invited to discuss any of these ideas, as well as propose your own. Feel free to [get familiar with the codebase](main#developers) already. To contact us, use one of these:

* [Tatoeba Wall page](http://tatoeba.org/wall/index)
* Email/Google groups: [Tatoeba GSoC mailing list](https://groups.google.com/forum/?fromgroups=#!forum/tatoeba-gsoc)
* IRC: [Tatoeba on #Freenode](irc://irc.freenode.net/tatoeba), [Webchat](http://webchat.freenode.net?channels=tatoeba)
* XMPP: [Tatoeba conference room on chat.tatoeba.org](xmpp:tatoeba@chat.tatoeba.org?join)


Current site
------------

Extending current PHP site: 

* programming in PHP with the CakePHP framework
* shell tools for maintenance (e.g.: better export scripts)
* JavaScript

### Better export scripts

Currently CSV dumps are done weekly. They require the database to be switched into a read-only mode, take 5~10 minutes, and do not contain some important information, such as tag creator and comments. CSV dumps are important for people who cooperate with Tatoeba by creating additional tools, so their quality is vital for healthy collaboration.

**Deliverables**: An database export mechanism that:

  * Dumps all interesting information (everything currently in the data dumps plus modification history, sentence comments, the wall, etc.).
  * Can create incremental dumps, making them faster, which will allow them to be made more frequently.
  * Provides an interface for collaborators to get notifications about new dumps and allows automatic access.
  * (Advanced) Provides a stream of updates via web sockets or a similar mechanism.

**Prerequisite knowledge**: a scripting language (Python preferred), PHP, MySQL.

### Administrative Scripts

The current site has experienced a couple of crashes and instabilities. We would like to increase the number of users who are fully capable of administering it with ease and reliability. In order to ease administration and quickly recover from disaster, a number of scripts covering common administrative tasks are needed.

**Deliverables**:

*Shell scripts* that cover:

 * backup
 * restoring of backups
 * export of data
 * import of data 
 * adding new languages
 * deduplication of sentences
 * indexing of sentences by the search daemon
 * preserving existing translations when typos in the source UI strings are fixed 
 * getting external services up and running
 * updating the production site from the repository
 * deployment on a real server from scratch
 * deployment on a development machine from scratch
 * monitoring the server and logging load and activity
 * other necessary tasks

An *administrative interface* accessible by admins to manually execute any of these tasks on the server with progress bars and statistics. 

**Prerequisite knowledge**: a scripting language (bash, Python, Perl, etc..), possibly familiarity with a build system (ansible, vagrant), and possibly familiarity with setting up and maintaining a monitoring system (newrelic, nagios, cacti, munin)

### Testing Suite

The current website doesn't have any tangible automated tests or any form of continuous integration. Any new code that gets added to the repository can break the existing website and most testing is done manually at this point. In order to make the code future proof and give the users a stable experience, a battery of tests for the main functionality of the website is needed.

**Deliverables**: A number of tests that create a test database and emulate a browser, testing some or all of the functionality that the website currently offers. Refer to this [list](http://en.wiki.tatoeba.org/articles/show/functionality-test-list)

**Prerequisite knowledge**: a scripting language (Python, PHP, etc...), MySQL, and a web testing framework (selenium, or something similar).

### API

The Tatoeba database is used either through the main website interface or through data dumps. Having a real API that can be called through AJAX and return machine-readable results would provide real-time access for external applications.

**Deliverables**: A web application that provides a set of API calls for data stored in the current database. The API should cover all data available through the current web interface, including sentence comments, wall comments, recently-added sentences and top recent contributors.

**Prerequisite knowledge**: a web application language (Python or PHP preferred), MySQL, and a data exchange format such as JSON or XML.

### API Compliant Interface

With the presence of an API and an API [spec](https://github.com/trang/tatoeba-api/wiki/Tatoeba-API-specification-2) most direct SQL queries in the current codebase will become obsolete and the preferred way to do it would be through the API. So a rewrite or a completely new Interface will be the next logical step.

**Deliverables**: A rewrite of the current codebase to use the API instead of SQL queries or a completely new interface in another language, preferably in a javascript framework.

**Prerequisite knowledge**: strong knowledge of CakePHP, familiarity with the API, and familiarity with the current codebase, or strong knowledge of a web framework (django for example) or a javascript framework (angularjs for example).

### Improvements in user interface for end users

Tatoeba now handles several kinds of queries, but more are desired. The translation interface, in particular, needs improvement. Examples of desired types of queries:

* Get all sentences in a given language by a given user that have not been translated into a given language. For example: *Show me all English sentences by user "CK" not yet translated into Japanese.*

* Same as above, but limited to sentences with audio. For example: *Show me all English sentences by "CK" with audio that have not been translated into Japanese.*

* Get all sentences by native speakers of a given language not yet translated into a given language. For example: *Show me all Finnish sentences by native speakers not translated into Hungarian.*

* Get all sentences in a given language with a certain tag not translated into a given language. For example: *Show me all Georgian sentences with the tag "restaurant" not translated into Armenian.*

* Same as above, but limited to sentences by native speakers not translated into a given language. For example: *Show me all Korean sentences by native speakers with the tag "weather" not translated into Japanese.*

* Get all sentences in a given language under a certain length not yet translated into a given language. For example: *Show me all Japanese sentences fewer than 50 characters in length not translated into French.*

* Same as above, but limited to native speaker sentences.

* Same as above, but limited to sentences by a given user.

* Get all sentences by native speakers of a given language that match a given search keyword that have not been translated into a given language. For example: *Show all English sentences with the word "mountain" not translated into Japanese.*

* Same as above, but limited to native speaker sentences.

* Same as above, but limited to sentences by a given user.

**Deliverables:** Implementation of some or all of the above. Project might include additional queries. It would be highly desired to provide a generic way of adding new types of queries.

**Prerequisite knowledge**: PHP, CakePHP.

### Allow users to follow each other

This feature would allow users to keep track of newly created sentences of other users, just like Twitter does. One could be able to get notified of new sentences of users he or she's following, and to browse them. Public and private visibility of who’s following who should be discussed prior to development.

**Deliverables:** a mean to follow one or more users ; a page that displays the sentences of the followed users and allows to browse and search through ; configurable notifications about new sentences of the followed users ; displaying of who’s following who.

**Prerequisite knowledge**: PHP, CakePHP.

### Word requests

This feature would allow users to request example sentences that show the correct usage of a given word or phrase. Contributors could browse lists of 'requested words' and add sentences that include them.

People interested in this idea should consider and discuss possible implementations details prior to development. Typical questions include: 

* What should be the scope of the lists (per-language, per-user…)? 
* How should the lists be maintained? 
* Can we indicate that a 'requested word' now has enough example sentences? If yes, how? 
* What’s the lifecycle of a typical requested word? 
* What if users want to express additional information in their requests, such as the context or sense for the requested word? 
* What about synonyms, inflections… ?

**Deliverables:** a mean to express the need of example sentences of a given word ; a way to easily contribute new sentences that shows example of wanted words

**Prerequisite knowledge**: PHP, CakePHP.

### Show pronunciation in IPA for sentences

IPA stands for "International Phonetic Alphabet" and is used to describe pronunciation of human languages in an unambiguous way. As such, it helps learning languages whose pronunciation rules are complex (e.g., English). Tatoeba could display IPA pronunciation for each sentence in basically the same way it currently displays pronunciation for Japanese using kana. One possible way of performing the task is to use an external library or application to prepare IPA annotations. For example, [eSpeak](http://espeak.sourceforge.net/) seems to be able to handle several popular languages and has an IPA converter.

**Deliverables:** A mechanism that shows IPA pronunciation for some languages (chosen by the student). This can be done server-side (as a standalone service or part of existing code) or client-side (using JavaScript). Mechanism should allow pre-generating pronunciation descriptions and should provide means to manually edit pronunciation later. Mechanism can rely on 3rd party tool to generate pronunciation descriptions.

**Prerequisite knowledge**: web technology, some web application stack (PHP, Python or CppCMS preferred).

New site & CppCMS
-----------------

Helping Sysko with tatowiki, tatodb. Extending [CppCMS](http://cppcms.com/wikipp/en/page/main). As the new site is still mostly being planned, there are no specific project ideas for the moment. Please ask on the IRC channel for more information. Note that many projects in this category will have an experimental nature, and their scope highly depends on your skills.

Standalone user tools
---------------------

Work on tools such as the following:

* [shtooka recorder](http://a4esl.com/temporary/tatoeba/shtooka/) or [swac-record](http://zmoo.fr/swac-tools/) for recording audio for sentences
* [tatoparser](https://github.com/qdii/tatoeba_parser) for retrieving sentences that match regular expressions
* [katoeba](https://github.com/sadhen/katoeba) 

Create new tools, such as apps for smartphones, for ordinary or advanced contributors.

### Android/iPhone application

iPhone users are about 12%, and Android users about 7% of the site visitors. It might help them immensely if they could use a dedicated application.

**Deliverables**: A smartphone application for easy access to Tatoeba. Examples of features:

* Querying the online Tatoeba site
* Adding sentences
* Translating
* Performing typical corpus maintenance tasks (linking/unlinking sentences, changing sentence language, tagging, etc.)
* Access to wall and sentence comments
* Recording voice
* Offline database access (more difficult!)

Note: It is not expected to implement all of these features during a single GSoC event. Depending on your skills, you might prepare a proposal for a basic set of features (if you don't have much experience in mobile development yet) or a more complex or targeted application (if you do have experience and want to prepare something more feature-complete).

**Prerequisite knowledge**: Java and Android development or iPhone and iOS developement; using web services.

### Streamlined linking of multiple sentences

Where multiple sentences in a source language have the same translation in the target language, make it easy to link those source sentences to the same target translation. Collecting the sentences that are likely to have the same translation could be as simple as presenting sentences in order of creation, since variants of a sentence that vary only in, e.g., the number of the pronoun (where the singular and plural forms of the second person map to the same word in English) are likely to be entered consecutively.

**Prerequisite knowledge**: JavaScript, possibly Java, possibly SQL

### Help bots

Produce bots like [those in Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Bots), to help with maintenance and repetitive tasks such as fixing common mistakes, wrong flag, etc. As on Wikipedia, users that are actually bots should be identified somehow on the website side. Ideally, create a library that could be used as a base to create bots that interact with the Tatoeba website.

Note: this idea would highly benefit from having a real API, which is another project listed here.

**Prerequisite knowledge**: web 

Other ideas
-----------

External services, like [CK's Temporary Tatoeba site](http://a4esl.com/temporary/tatoeba/). Other ideas.

### XMPP Integration for Tatoeba

Use the XMPP communications protocol to integrate the manipulation of sentences, comments, and wall posts, as well as live feeds of latest comments, sentence additions, and wall posts, with services such as pubsub.

**Deliverables**:

* An XEP that outlines the protocol tatoeba would use for all of those operations over XMPP ready to be submitted to the XSF
* An implementation of this XEP in XMPP clients as plugins, poezio and gajim are top priorities.
* An implementation of this XEP server side as a module, prosody is a top priority.

**Prerequisite Knowledge**: XMPP, PubSub, Python, Lua, Familiarity with prosody/gajim/poezio codebases and plugin architecture


### SRS deck generator

Spaced Repetition Systems such as Anki and Mnemosyne are popular tools for learning languages. However, preparing a good SRS deck is a time-consuming task. Therefore, an automated way to generate a deck from a list of sentences (e.g., sentences on a Tatoeba list, sentences tagged by some specific tag, etc.) would help language learners.

**Deliverables**: an application (preferably a web-based one) that would use Tatoeba database (for example in the form of a weekly CSV data dump) to create SRS decks for major flash card applications. Examples of features:

* Generate a simple deck from a Tatoeba list, tag, or search query.
* Generate an N+1-style deck based on user's list of known words and Tatoeba database. (User gives a list of N words that s/he already knows. System chooses a new sentence where exactly one word is unknown, and the rest belong to the already known set.)
* Generated decks have proper internal structure (as for Anki decks: proper field scheme is used to store knowledge, so editing is easy).

**Prerequisite knowledge**: any web stack, however Django or CppCMS are prefered; Python or C++; knowledge about SRS.

### Browsable graph of sentence links

Given a sentence, display it as a [graph](https://en.wikipedia.org/wiki/Graph_%28data_structure%29) [like this](http://blog.tatoeba.org/2010/02/how-to-be-good-contributor-in-tatoeba.html#rule2) the linked sentences up to a given depth. The main purpose of such a graph is to show users how Tatoeba is structured at a glance. The current interface doesn’t provide such a view, but it’s important that users understand the actual structure of Tatoeba. This idea could be freely extended to a complete interface allowing linking and unlinking with a click, filter by language, edit sentences, or whatever you can think of.

**Deliverables**: a web application or a client-side JavaScript program that provides a graph view of a group of sentences, and allows manipulating them. Code can either operate on database directly or use existing or planned APIs.

Note that this idea can be implemented as part of the current code base in PHP or as an experimental service for a new CppCMS site (preferred).

**Prerequisite knowledge**: PHP, Python or CppCMS.

version at: 02/03/2014, 18:38

GSoC ideas for student projects
===============================

This page lists example ideas for students who would like to take part in [Google Summer of Code](http://www.google-melange.com/) and be mentored by Tatoeba. To quote [GSoC FAQ](http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2013/help_page#3._What_is_an_Ideas_list):

<blockquote>
<p>An Ideas list should be a list of suggested student projects. This list is meant to introduce contributors to your project's needs and to provide inspiration to would-be student applicants. It is useful to classify each idea as specifically as possible, e.g. "must know Python" or "easier project; good for a student with more limited experience with C++." If your organization plans to provide a proposal template for the students, it would be good to include it on your Ideas list.</p>

<p>Keep in mind that your ideas list should be a starting point for student proposals; we've heard from past mentoring organization participants that some of their best student projects are those that greatly expanded on a proposed idea or were blue-sky proposals not mentioned on the ideas list at all. A link to a bug tracker for your open source organization is NOT an ideas list.</p>

<p>You can check out the <a href="http://community.kde.org/GSoC/2011/Ideas">Ideas list for KDE</a> for Google Summer of Code in 2011 to get an idea of what we’re looking for in an ideas list. </p>
</blockquote>

If you're a student, you're invited to discuss any of these ideas, as well as propose your own. Feel free to [get familiar with the codebase](main#developers) already. To contact us, use one of these:

* [Tatoeba Wall page](http://tatoeba.org/wall/index)
* Email/Google groups: [Tatoeba's dev mailing list](https://groups.google.com/forum/#!forum/tatoebaproject)
* IRC: [Tatoeba on #Freenode](irc://irc.freenode.net/tatoeba), [Webchat](http://webchat.freenode.net?channels=tatoeba)
* XMPP: [Tatoeba conference room on chat.tatoeba.org](xmpp:tatoeba@chat.tatoeba.org?join)


Current site
------------

Extending current PHP site: 

* programming in PHP with the CakePHP framework
* shell tools for maintenance (e.g.: better export scripts)
* JavaScript

### Better export scripts

Currently CSV dumps are done weekly. They require the database to be switched into a read-only mode, take 5~10 minutes, and do not contain some important information, such as tag creator and comments. CSV dumps are important for people who cooperate with Tatoeba by creating additional tools, so their quality is vital for healthy collaboration.

**Deliverables**: An database export mechanism that:

  * Dumps all interesting information (everything currently in the data dumps plus modification history, sentence comments, the wall, etc.).
  * Can create incremental dumps, making them faster, which will allow them to be made more frequently.
  * Provides an interface for collaborators to get notifications about new dumps and allows automatic access.
  * (Advanced) Provides a stream of updates via web sockets or a similar mechanism.

**Prerequisite knowledge**: a scripting language (Python preferred), PHP, MySQL.

### Administrative Scripts

The current site has experienced a couple of crashes and instabilities. We would like to increase the number of users who are fully capable of administering it with ease and reliability. In order to ease administration and quickly recover from disaster, a number of scripts covering common administrative tasks are needed.

**Deliverables**:

*Shell scripts* that cover:

 * backup
 * restoring of backups
 * export of data
 * import of data 
 * adding new languages
 * deduplication of sentences
 * indexing of sentences by the search daemon
 * preserving existing translations when typos in the source UI strings are fixed 
 * getting external services up and running
 * updating the production site from the repository
 * deployment on a real server from scratch
 * deployment on a development machine from scratch
 * monitoring the server and logging load and activity
 * other necessary tasks

An *administrative interface* accessible by admins to manually execute any of these tasks on the server with progress bars and statistics. 

**Prerequisite knowledge**: a scripting language (bash, Python, Perl, etc..), possibly familiarity with a build system (ansible, vagrant), and possibly familiarity with setting up and maintaining a monitoring system (newrelic, nagios, cacti, munin)

### Testing Suite

The current website doesn't have any tangible automated tests or any form of continuous integration. Any new code that gets added to the repository can break the existing website and most testing is done manually at this point. In order to make the code future proof and give the users a stable experience, a battery of tests for the main functionality of the website is needed.

**Deliverables**: A number of tests that create a test database and emulate a browser, testing some or all of the functionality that the website currently offers. Refer to this [list](http://en.wiki.tatoeba.org/articles/show/functionality-test-list)

**Prerequisite knowledge**: a scripting language (Python, PHP, etc...), MySQL, and a web testing framework (selenium, or something similar).

### API

The Tatoeba database is used either through the main website interface or through data dumps. Having a real API that can be called through AJAX and return machine-readable results would provide real-time access for external applications.

**Deliverables**: A web application that provides a set of API calls for data stored in the current database. The API should cover all data available through the current web interface, including sentence comments, wall comments, recently-added sentences and top recent contributors.

**Prerequisite knowledge**: a web application language (Python or PHP preferred), MySQL, and a data exchange format such as JSON or XML.

### API Compliant Interface

With the presence of an API and an API [spec](https://github.com/trang/tatoeba-api/wiki/Tatoeba-API-specification-2) most direct SQL queries in the current codebase will become obsolete and the preferred way to do it would be through the API. So a rewrite or a completely new Interface will be the next logical step.

**Deliverables**: A rewrite of the current codebase to use the API instead of SQL queries or a completely new interface in another language, preferably in a javascript framework.

**Prerequisite knowledge**: strong knowledge of CakePHP, familiarity with the API, and familiarity with the current codebase, or strong knowledge of a web framework (django for example) or a javascript framework (angularjs for example).

### Improvements in user interface for end users

Tatoeba now handles several kinds of queries, but more are desired. The translation interface, in particular, needs improvement. Examples of desired types of queries:

* Get all sentences in a given language by a given user that have not been translated into a given language. For example: *Show me all English sentences by user "CK" not yet translated into Japanese.*

* Same as above, but limited to sentences with audio. For example: *Show me all English sentences by "CK" with audio that have not been translated into Japanese.*

* Get all sentences by native speakers of a given language not yet translated into a given language. For example: *Show me all Finnish sentences by native speakers not translated into Hungarian.*

* Get all sentences in a given language with a certain tag not translated into a given language. For example: *Show me all Georgian sentences with the tag "restaurant" not translated into Armenian.*

* Same as above, but limited to sentences by native speakers not translated into a given language. For example: *Show me all Korean sentences by native speakers with the tag "weather" not translated into Japanese.*

* Get all sentences in a given language under a certain length not yet translated into a given language. For example: *Show me all Japanese sentences fewer than 50 characters in length not translated into French.*

* Same as above, but limited to native speaker sentences.

* Same as above, but limited to sentences by a given user.

* Get all sentences by native speakers of a given language that match a given search keyword that have not been translated into a given language. For example: *Show all English sentences with the word "mountain" not translated into Japanese.*

* Same as above, but limited to native speaker sentences.

* Same as above, but limited to sentences by a given user.

**Deliverables:** Implementation of some or all of the above. Project might include additional queries. It would be highly desired to provide a generic way of adding new types of queries.

**Prerequisite knowledge**: PHP, CakePHP.

### Allow users to follow each other

This feature would allow users to keep track of newly created sentences of other users, just like Twitter does. One could be able to get notified of new sentences of users he or she's following, and to browse them. Public and private visibility of who’s following who should be discussed prior to development.

**Deliverables:** a mean to follow one or more users ; a page that displays the sentences of the followed users and allows to browse and search through ; configurable notifications about new sentences of the followed users ; displaying of who’s following who.

**Prerequisite knowledge**: PHP, CakePHP.

### Word requests

This feature would allow users to request example sentences that show the correct usage of a given word or phrase. Contributors could browse lists of 'requested words' and add sentences that include them.

People interested in this idea should consider and discuss possible implementations details prior to development. Typical questions include: 

* What should be the scope of the lists (per-language, per-user…)? 
* How should the lists be maintained? 
* Can we indicate that a 'requested word' now has enough example sentences? If yes, how? 
* What’s the lifecycle of a typical requested word? 
* What if users want to express additional information in their requests, such as the context or sense for the requested word? 
* What about synonyms, inflections… ?

**Deliverables:** a mean to express the need of example sentences of a given word ; a way to easily contribute new sentences that shows example of wanted words

**Prerequisite knowledge**: PHP, CakePHP.

### Show pronunciation in IPA for sentences

IPA stands for "International Phonetic Alphabet" and is used to describe pronunciation of human languages in an unambiguous way. As such, it helps learning languages whose pronunciation rules are complex (e.g., English). Tatoeba could display IPA pronunciation for each sentence in basically the same way it currently displays pronunciation for Japanese using kana. One possible way of performing the task is to use an external library or application to prepare IPA annotations. For example, [eSpeak](http://espeak.sourceforge.net/) seems to be able to handle several popular languages and has an IPA converter.

**Deliverables:** A mechanism that shows IPA pronunciation for some languages (chosen by the student). This can be done server-side (as a standalone service or part of existing code) or client-side (using JavaScript). Mechanism should allow pre-generating pronunciation descriptions and should provide means to manually edit pronunciation later. Mechanism can rely on 3rd party tool to generate pronunciation descriptions.

**Prerequisite knowledge**: web technology, some web application stack (PHP, Python or CppCMS preferred).

New site & CppCMS
-----------------

Helping Sysko with tatowiki, tatodb. Extending [CppCMS](http://cppcms.com/wikipp/en/page/main). As the new site is still mostly being planned, there are no specific project ideas for the moment. Please ask on the IRC channel for more information. Note that many projects in this category will have an experimental nature, and their scope highly depends on your skills.

Standalone user tools
---------------------

Work on tools such as the following:

* [shtooka recorder](http://a4esl.com/temporary/tatoeba/shtooka/) or [swac-record](http://zmoo.fr/swac-tools/) for recording audio for sentences
* [tatoparser](https://github.com/qdii/tatoeba_parser) for retrieving sentences that match regular expressions
* [katoeba](https://github.com/sadhen/katoeba) 

Create new tools, such as apps for smartphones, for ordinary or advanced contributors.

### Android/iPhone application

iPhone users are about 12%, and Android users about 7% of the site visitors. It might help them immensely if they could use a dedicated application.

**Deliverables**: A smartphone application for easy access to Tatoeba. Examples of features:

* Querying the online Tatoeba site
* Adding sentences
* Translating
* Performing typical corpus maintenance tasks (linking/unlinking sentences, changing sentence language, tagging, etc.)
* Access to wall and sentence comments
* Recording voice
* Offline database access (more difficult!)

Note: It is not expected to implement all of these features during a single GSoC event. Depending on your skills, you might prepare a proposal for a basic set of features (if you don't have much experience in mobile development yet) or a more complex or targeted application (if you do have experience and want to prepare something more feature-complete).

**Prerequisite knowledge**: Java and Android development or iPhone and iOS developement; using web services.

### Streamlined linking of multiple sentences

Where multiple sentences in a source language have the same translation in the target language, make it easy to link those source sentences to the same target translation. Collecting the sentences that are likely to have the same translation could be as simple as presenting sentences in order of creation, since variants of a sentence that vary only in, e.g., the number of the pronoun (where the singular and plural forms of the second person map to the same word in English) are likely to be entered consecutively.

**Prerequisite knowledge**: JavaScript, possibly Java, possibly SQL

### Help bots

Produce bots like [those in Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Bots), to help with maintenance and repetitive tasks such as fixing common mistakes, wrong flag, etc. As on Wikipedia, users that are actually bots should be identified somehow on the website side. Ideally, create a library that could be used as a base to create bots that interact with the Tatoeba website.

Note: this idea would highly benefit from having a real API, which is another project listed here.

**Prerequisite knowledge**: web 

Other ideas
-----------

External services, like [CK's Temporary Tatoeba site](http://a4esl.com/temporary/tatoeba/). Other ideas.

### XMPP Integration for Tatoeba

Use the XMPP communications protocol to integrate the manipulation of sentences, comments, and wall posts, as well as live feeds of latest comments, sentence additions, and wall posts, with services such as pubsub.

**Deliverables**:

* An XEP that outlines the protocol tatoeba would use for all of those operations over XMPP ready to be submitted to the XSF
* An implementation of this XEP in XMPP clients as plugins, poezio and gajim are top priorities.
* An implementation of this XEP server side as a module, prosody is a top priority.

**Prerequisite Knowledge**: XMPP, PubSub, Python, Lua, Familiarity with prosody/gajim/poezio codebases and plugin architecture


### SRS deck generator

Spaced Repetition Systems such as Anki and Mnemosyne are popular tools for learning languages. However, preparing a good SRS deck is a time-consuming task. Therefore, an automated way to generate a deck from a list of sentences (e.g., sentences on a Tatoeba list, sentences tagged by some specific tag, etc.) would help language learners.

**Deliverables**: an application (preferably a web-based one) that would use Tatoeba database (for example in the form of a weekly CSV data dump) to create SRS decks for major flash card applications. Examples of features:

* Generate a simple deck from a Tatoeba list, tag, or search query.
* Generate an N+1-style deck based on user's list of known words and Tatoeba database. (User gives a list of N words that s/he already knows. System chooses a new sentence where exactly one word is unknown, and the rest belong to the already known set.)
* Generated decks have proper internal structure (as for Anki decks: proper field scheme is used to store knowledge, so editing is easy).

**Prerequisite knowledge**: any web stack, however Django or CppCMS are prefered; Python or C++; knowledge about SRS.

### Browsable graph of sentence links

Given a sentence, display it as a [graph](https://en.wikipedia.org/wiki/Graph_%28data_structure%29) [like this](http://blog.tatoeba.org/2010/02/how-to-be-good-contributor-in-tatoeba.html#rule2) the linked sentences up to a given depth. The main purpose of such a graph is to show users how Tatoeba is structured at a glance. The current interface doesn’t provide such a view, but it’s important that users understand the actual structure of Tatoeba. This idea could be freely extended to a complete interface allowing linking and unlinking with a click, filter by language, edit sentences, or whatever you can think of.

**Deliverables**: a web application or a client-side JavaScript program that provides a graph view of a group of sentences, and allows manipulating them. Code can either operate on database directly or use existing or planned APIs.

Note that this idea can be implemented as part of the current code base in PHP or as an experimental service for a new CppCMS site (preferred).

**Prerequisite knowledge**: PHP, Python or CppCMS.

Note

The lines in green are the lines that have been added in the new version. The lines in red are those that have been removed.