https://github.com/programarivm/babylon
You'll learn about how programs can perform tasks by feeding them data rather than writing explicit code.
The following two posts will provide you with some context to follow today's tutorial.
Dear readers, how are you today? I wonder if you'd like to help me feed an intelligent language detector that can learn new languages very easily.
In case you missed my previous post on machine learning with PHP, let me tell you that Babylon 0.9.1 is already working okay with a bunch of ISO 8859 languages:
| ISO 639-3 Code | Language |
|---|---|
ces | Czech |
cym | Welsh |
dan | Danish |
deu | German |
eng | English |
fin | Finnish |
fra | French |
gla | Scottish Gaelic |
gle | Irish |
hun | Hungarian |
ita | Italian |
isl | Icelandic |
nld | Dutch; Flemish |
nob | Norwegian |
pol | Polish |
por | Portuguese |
ron | Romanian |
spa | Spanish |
swe | Swedish |
tgl | Tagalog |
Note: Currently Babylon supports ISO 8859 languages only, which are those ones using some form of latin alphabet -- English, Italian, German, Tagalog, Turkish are a few examples among many others.
Can you imagine? There are dozens of different ISO 8859 languages out there! So your help is really appreciated.
Let me show you with an example how you could help me teach Babylon new languages. I'm going to replicate the steps that I followed to teach it Cebuano.
First of all, I did some digging on the Internet and found a public domain ebook written in Cebuano.
Project Gutenberg is a nice resource for this purpose since it contains a plethora of ebooks in different languages. However I must say it does not support all languages in the world. There are no ebooks in Turkish for example, which is a bit of a shame keeping in mind this language has around 65 million speakers.
Be that as it may, as I said before we are teaching Babylon the Cebuano language. So let's just pick an ebook in Cebuano and copy and paste its content into the babylon/dataset/input/iso-8859/latin/austronesian/ceb.txt file.
Did you know that Cebuano is an Austronesian language?
As you see, we're following a convention consisting of naming the files with their ISO 639-3 code counterpart.
Also make sure to remove all English words from the beginning and the end of the file as it is shown in the example above. This is because the core idea implemented in Babylon consists in computing the most frequent words in this or that language, and we want to remove bias from our data sets.
Then, I run the command:
php cli/prepare.php
This will create a CSV with the most frequent words in all of the files in the dataset/input folder.
The operation may take a few seconds to be completed.
Do you want to proceed? (Y/N): y
OK! The most frequent words in ceb.txt were transformed into CSV format...
OK! The most frequent words in tgl.txt were transformed into CSV format...
The austronesian language family has been updated.
OK! The most frequent words in cym.txt were transformed into CSV format...
OK! The most frequent words in gla.txt were transformed into CSV format...
OK! The most frequent words in gle.txt were transformed into CSV format...
The gaelic language family has been updated.
OK! The most frequent words in dan.txt were transformed into CSV format...
OK! The most frequent words in deu.txt were transformed into CSV format...
OK! The most frequent words in eng.txt were transformed into CSV format...
OK! The most frequent words in isl.txt were transformed into CSV format...
OK! The most frequent words in nld.txt were transformed into CSV format...
OK! The most frequent words in nob.txt were transformed into CSV format...
OK! The most frequent words in swe.txt were transformed into CSV format...
The germanic language family has been updated.
OK! The most frequent words in fra.txt were transformed into CSV format...
OK! The most frequent words in ita.txt were transformed into CSV format...
OK! The most frequent words in por.txt were transformed into CSV format...
OK! The most frequent words in ron.txt were transformed into CSV format...
OK! The most frequent words in spa.txt were transformed into CSV format...
The romance language family has been updated.
OK! The most frequent words in ces.txt were transformed into CSV format...
OK! The most frequent words in pol.txt were transformed into CSV format...
The slavic language family has been updated.
OK! The most frequent words in fin.txt were transformed into CSV format...
OK! The most frequent words in hun.txt were transformed into CSV format...
The uralic language family has been updated.
OK! The words in austronesian.csv were successfully read...
OK! The words in gaelic.csv were successfully read...
OK! The words in germanic.csv were successfully read...
OK! The words in romance.csv were successfully read...
OK! The words in slavic.csv were successfully read...
OK! The words in uralic.csv were successfully read...
OK! iso-8859-latin-family.csv was successfully written...
Operation completed.
That's it!
The prepare.php command calculates the most frequent words in all languages storing the results in the babylon/dataset/output/ folder. On the other hand, the babylon/dataset/output/iso-8859-latin-family.csv file, which holds disjointish sets of words, is kind of the language families' digital fingerprint.
With this statistics, now Babylon can detect the Cebuano language as it is shown next.
php cli/detect/language.php "Kong kini madangat, ang mga bana hayan mangaklas tungud kay dili makaantus sa paghikay sa abó ug pagpanakdak sa mga labhanan."
ceb
Finally let me suggest it is a good idea to write some tests to make sure the Cebuano language detection won't break in future releases for any unexpected reason we just can't foresee right now.
// tests/unit/AustronesianTest.php
...
/**
* @dataProvider cebData
* @test
*/
public function family_detect_ceb($text)
{
$this->assertEquals('austronesian', (new FamilyDetector($text))->detect());
}
...
public function cebData()
{
return [
[
"Sa usá ka bangko, naglingkod ang duhá ka tawo. Babaye ang usá
nga nagmaskará ug lalake ang ikaduhá nga waláy maskará. Ang lake
nagkanayón:"
],
];
}
Let's recap. Here is all we did to add a new ISO 8859 language:
babylon/dataset/input/iso-8859/latin/ folderphp cli/prepare.phpInstead of writing PHP code, we've just followed a straightforward three-steps process.
For more detailed information on how to add a new language, please visit the Cebuano language is taught to Babylon #14. Click on GaelicTest passes #12 to learn how to add a new family of languages.
I hope you liked today's post! Thank you for reading and sharing your thoughts.