DictionaryForMids Forum

Hi, Im a developer of Apertium, an open-source platform for developing rule-based machine translation systems.

Ive also just got an Android phone and saw your palette of dictionaries downloadable directly thru the Android version. Quite impressive, but a lot of languages are missing.

We have ~20 bilingual dictionaries (see http://wiki.apertium.org/wiki/Main_Page) available on GPL.
Would you be interested in making these dictionaries available?

There have also been some work on http://lernu.net/ to make its big collection of Esperanto dictionaries available (see http://en.lernu.net/helpo/vortaroj.php and expand the VORTARO widget at the right).
If successfull, could these also appear inside the Android application?


Yours,
Jacob

Jacob,

well, of course we are interested to get your dictionaries on DictionaryForMIDs :D (even though until now I did not yet well understand the mechanisms of Apertium  :-\ ).

Cause your dictionaries are available under GPL we can put them on the DictionaryForMIDs web site for public download.

Concerning the list of dictionaries for Android: the Android version as well as the list of dictionaries for Android is maintained by Achim. Achim will have to speak on this.

Best regards,
Gert

Hi Jacob, hi Gert,

Quote from: Gert on 23. April 2010, 19:25:57
Cause your dictionaries are available under GPL we can put them on the DictionaryForMIDs web site for public download.
Concerning the list of dictionaries for Android: the Android version as well as the list of dictionaries for Android is maintained by Achim. Achim will have to speak on this.

As soon as the dictionaries are available on the DictionaryForMIDs website I add the new links to the OTA installation system. They'll then immediately be available for automatic installation on Android devices. (You may need to drop me a quick note or it may take some time for me to realize that there are new files available...)

It's that easy  8)

Cheers,
Achim

This is great news. I'll proceed to generate some dictionary files from Apertium data, then.

WRT "understand the mechanisms of Apertium" it would take some time - it's a full fledged MT system, you have to understand some linguistics to dive in. But ask, and I will happily explain :-)


Jacob

OK, here is first try:

Here is what I did:

lt-expand /home/j/esperanto/apertium/apertium-eo-en/apertium-eo-en.eo-en.dix > input0.txt

cat input0.txt | grep -v ':>:' | grep -v ':<:' | sed -re 's/<[^:]+:/:/g' | grep -v '><' | grep -v REGEX | grep -v '#' | sed -re "s/:/\t/g" | sed -e 's/</[/g' | sed -e 's/>/]/g' > input1.txt

rm -rf outputdictionary
mkdir outputdictionary

java -jar ../DictionaryGeneration.jar input1.txt outputdictionary .


The whole stuff (input and result) is here: http://javabog.dk/filer/en-eo.zip


However, I can't run it, as it seems your PC version (http://dictionarymid.sourceforge.net/pc.html) doesent support seem to work for me (on Linux). 
I tried DictionaryForMIDs_2.4.0_EngPor_IDP_dev.zip and DictionaryForMIDs_3.4.0_IDP(Eng-Fra).zip
I get "The dictionary uses own programcode and may not work as expected."

$ java -jar DictionaryForMIDs/DictionaryForMIDs.jar
Exception in thread "main" java.lang.NullPointerException
   at de.kugihan.dictionaryformids.hmi_j2se.DictionaryForSE.fillTableColums(DictionaryForSE.java:352)
   at de.kugihan.dictionaryformids.hmi_j2se.DictionaryForSE.<init>(DictionaryForSE.java:120)
   at de.kugihan.dictionaryformids.hmi_j2se.DictionaryForSE.main(DictionaryForSE.java:98)


Will I have to actually install it on my Android to be able to see of it works?

You need to run JarCreator in order to build the files DictionaryForMIDs_xxx.jar and DictionaryForMIDs.jad (actually, the jad-file is only need for Java ME).

Also for the PC version you need to have the file DictionaryForMIDs_xxx.jar; there you can simple load this Jar-file.

Best regards,
Gert

Quote from: jn0101 on 25. April 2010, 00:44:19
OK, here is first try:

Here is what I did:

Code Select Expand


lt-expand /home/j/esperanto/apertium/apertium-eo-en/apertium-eo-en.eo-en.dix > input0.txt

cat input0.txt | grep -v ':>:' | grep -v ':<:' | sed -re 's/<[^:]+:/:/g' | grep -v '><' | grep -v REGEX | grep -v '#' | sed -re "s/:/\t/g" | sed -e 's/</[/g' | sed -e 's/>/]/g' > input1.txt


Btw, you got the languages mixed up: language1 seems to be Esperanto...

Good work though, I tried on Android and it looks fine!

Achim

Great to hear you liked it. I'm really happy for the positive feedback.


Here is a 2nd try: http://javabog.dk/filer/2nd_try.zip

In this file I have Esperanto-English and Esperanto-Nepali.
The dictionary files are in eo-en/jar/dictionary and in eo-ne/jar/dictionary/.

To be able to test Ive also packaged using an existing package with JAR/JAD (DictionaryForMIDs_3.4.0_IDP(Eng-Fra).zip). I think your website is great but I feel you lack a wiki where users can document themselves how to get the things up and running. F.ex I have been in doubt how I could most easily test my work (I couldnt get the PC version to work, but Ive managed to get a midlet running in an emulator, so no worry).

Here are some questions:

1) Esperanto have some special characters (ĉ, ĝ, ĵ ...). For example a horse is 'ĉevalo'. Would it be possible that user enters e.g. 'cevalo' (w.o the ^), 'cxevalo' or '^cevalo' for searching such words?
Has this something to do with language1NormationClassName ? Will I have to write my own class to get support for this?


2) Nepali is based on the Devanagari alphabet. How to handle searching in that?
(BTW I did some transliteration work in Java - the project is dead but the code remains: http://code.google.com/p/nepaliconverter)


3) General customization.
It seems MIDlet-Vendor: must be 'Gert Nuber and Contributors'. I'd like to have something like 'Data from the Apertium Project'
Where to put the GPL license/copyright notice/source ?


4) Semisynonyms and directions. In Apertium entries can be in one direction (i.e. you have a A==B and an unidirectional A2->B, where A2 is a noncommon synonym for A). Should such things be handled in DfM ?


5) Naming conventions for word classes. I see you usually use e.g. [Noun]. I am unsure to what range I should stick to your naming conventions. In eo-ne/konverti.sh Ive done like this:

| sed 's/<n>/ [Nomo]/g' \
| sed 's/<np>/ [Propra nomo]/g' \
| sed 's/<adj>/ [Adjektivo]/g' \
| sed 's/<adv>/ [Adverbo]/g' \
| sed 's/<cnjcoo>/ [Konjunkcio]/g' \
| sed 's/<prn>/ [Pronomo]/g' \
| sed 's/<num>/ [Numero]/g' \
| sed 's/<vblex><itr>/ [Verbo netransitiva]/g' \
| sed 's/<vblex><tr>/ [Verbo transitiva]/g' \
| sed 's/<vblex><ditr>/ [Verbo duoble transitiva]/g' \
| sed 's/<ij>/ [Interjekcio, krivorto]/g' \

(assuming you know the Unix command 'sed'... or are you all Windows guys? ;-)


6) It would be great with some collaborative setup where you could edit the files, to fix stuff. Is there somewhere where we could share files (Apertium has SVN, if I press them I can get you a SVN write access, but perhaps you have an easier option... I don't know how strict you police your SVN access).


7) I have an Esperanto translation of the UI lying around. I'll send it when I get it digged out. Does the Android version use the same localization strings?



Thank you,
and Achim, if you could send me a sample Android app / data, I'd be thankfull (or is it just to ZIP the stuff and put on the SD card and open it?)

Wow, a lot of questions, hope its OK :-)

Yours,
Jacob

QuoteWow, a lot of questions, hope its OK :-)

Sure, that is why we have a forum !!


I am short of time right now (and I will not be able to read emails the next week), so I will just hastily try to address one or the other of your questions. Fortunately we do have active supporters for DfM who may provide you all the support you need  ::)

Quote1) Esperanto have some special characters (ĉ, ĝ, ĵ ...). For example a horse is 'ĉevalo'. Would it be possible that user enters e.g. 'cevalo' (w.o the ^), 'cxevalo' or '^cevalo' for searching such words?
Has this something to do with language1NormationClassName ? Will I have to write my own class to get support for this?

If the characters are based on the Latin characters, then NormationLat should do. NormationLat will allow you to search, e.g., "ĉ" with a normal "c".
Please have a look at NormationLat.java to check whether all characters are covered.

Quote3) General customization.
It seems MIDlet-Vendor: must be 'Gert Nuber and Contributors'. I'd like to have something like 'Data from the Apertium Project'
Where to put the GPL license/copyright notice/source ?

There are two copyrights:
(1) for the dictionary application (which is for Java ME: 'Gert Nuber and Contributors')
and
(2) for the dictionary data; you need to put that in the infoText-property (see http://dictionarymid.sourceforge.net/newdict.html (http://dictionarymid.sourceforge.net/newdict.html)), additionally you may include a detailed readme/license in the ZIP file.

Quote4) Semisynonyms and directions. In Apertium entries can be in one direction (i.e. you have a A==B and an unidirectional A2->B, where A2 is a noncommon synonym for A). Should such things be handled in DfM ?

You can do either way in DfM. It just depends on the input that you provide to DictionaryGeneration, respectively, if you are really much advanced, you could use a DictionaryUpdate-class for special processing. In general, DictionaryForMIDs provides a lot of flexibility for these things. We should look at your item 4) closely to figure out how to do this best.

Quote5) Naming conventions for word classes. I see you usually use e.g. [Noun]. I am unsure to what range I should stick to your naming conventions. In eo-ne/konverti.sh Ive done like this:

If the predefined contents do not match your needs, just use your own ones; no problem with that.

Quote7) I have an Esperanto translation of the UI lying around. I'll send it when I get it digged out. Does the Android version use the same localization strings?

DfM and Android both have their own User Interface and as such their own UI translation files. I will gladly incorporate the Esperanto translation for Java ME (guess Achim may do for Android).

@Achim: oh, what is the latest on the Java ME translations of the International Mother Language Day ... honestly speaking I did not yet work those into the Java ME version.

Regards,
Gert

WRT 7): Great Gert. Ive sent you the translation in a private mail.

Jacob

@Achim: Ive corrected the Danish translation (the æ sign had become an œ) and added a draft of an Esperanto translation.

Here is output of svn diff. My sourceforge username is 'nordfalk' if you prefer that I add it myself. The Esperanto translation will certainly needs updating, as its just a quick draft that Ive also sent to a translator.

Jacob

Achim:

Here is the final version of Android/res/values-eo/strings.xml

Yours,
Jacob

Hi Jacob,

thanks for your great update! Can you check the file Android/res/values-eo/strings.xml in your previous post again? It seems to be empty... Just upload it once more or send me an email, I'll push it together with your patch to the SVN.

To test a dictionary on Android, put the dictionary directory (the one with one DictionaryForMIDs.properties and many *.csv files) somewhere on the SD-card. You could also just put the jar file there, but speed will get significantly slower...

-Achim

Sure, here it is.

BTW Ive made changes in a few places in the source tree.

$ svn status
M       Android/res/values/app-strings.xml
M       Android/res/values/strings.xml
A       Android/res/values-eo
A       Android/res/values-eo/strings.xml
M       Android/res/values-da/strings.xml
M       Android/res/values-zh-rTW/strings.xml
M       Android/res/values-hu/strings.xml
M       Android/res/values-zh-rCN/strings.xml
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/lcdui_extension/DfMForm.java
M      DictionaryForMIDs
?       DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java


The change in JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/lcdui_extension/DfMForm.java is a workaround to use microemu.  See
http://code.google.com/p/microemu/issues/detail?id=44&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary%20Reporter%20Stars


DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java is Esperanto normalization.

Jacob

Try this:
Open emulator at http://microemu.org/microemu-webstart/index.html
Open http://javabog.dk/filer/paroj/eo-en/ and drag JAD file to it.

Hi Jacob,

Quote from: jn0101 on 28. April 2010, 14:18:27
Sure, here it is.

BTW Ive made changes in a few places in the source tree.

thanks, I just committed all the Android related files.

Quote from: jn0101 on 28. April 2010, 14:24:06
Try this:
Open emulator at http://microemu.org/microemu-webstart/index.html
Open http://javabog.dk/filer/paroj/eo-en/ and drag JAD file to it.

Hey, that would make a cool demo on our website!

-Achim

Quote from: jn0101 on 28. April 2010, 14:24:06

Try this:
Open emulator at http://microemu.org/microemu-webstart/index.html
Open http://javabog.dk/filer/paroj/eo-en/ and drag JAD file to it.

Or, just try this link: http://microemu.org/webstart/javabog.dk/filer/paroj/eo-en/DictionaryForMIDs_eo-en_lernu.jnlp

I really think the DfM home page should have such links to enable users to try out the application and dictionaries easily before they install on their phones!

What do you think?

Really, during the years, I have passed by the DfM home page many times, but just never got to install the app. I think a possibility for easily trying out your stuff would triple your user base!

Jacob

@Achim: Could you please apply this patch?

It contains:

1) Updates to include Esperanto in the J2ME GUI:
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/UIDisplayTextContents.java
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/UIDisplayTextItems.java
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/DictionaryForMIDs.languages


2) An update that will make DfM work with microemu.  See
http://code.google.com/p/microemu/issues/detail?id=44&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary%20Reporter%20Stars
The update should be deleted in 6 months (when a new version of microemu is released)

M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/lcdui_extension/DfMForm.java


3) This is most important:

A         DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java

I need this to be able to use the dictionaries.


If you could do a new Android build released to Market with 3) included I would be very gratefull, as I could instruct Esperanto speakers to download it and ZIP files with the dictionaries.
If not, could you do a build and instruct me on how to install it on other user's phones?


Thanks,
Jacob

Hi Jacob

Quote from: jn0101 on 29. April 2010, 16:11:00
3) This is most important:

A         DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java

If you could do a new Android build released to Market with 3) included I would be very gratefull, as I could instruct Esperanto speakers to download it and ZIP files with the dictionaries.
If not, could you do a build and instruct me on how to install it on other user's phones?

I just committed the normation and pushed the updated version 0.12 (including Esperanto localization) to the Market.

Currently I don't have the JavaME source set up, but I guess Gert will have look as soon as he has a minute (@Gert...).
Could you post some steps on how to support microemu and how to create the jnlp files? Or refer me to the relevant documentation?

I really appreciate your work!
Achim

Quote from: axin on 30. April 2010, 10:02:47
Hi Jacob

Quote from: jn0101 on 29. April 2010, 16:11:00
3) This is most important:

A         DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java

If you could do a new Android build released to Market with 3) included I would be very gratefull, as I could instruct Esperanto speakers to download it and ZIP files with the dictionaries.
If not, could you do a build and instruct me on how to install it on other user's phones?

I just committed the normation and pushed the updated version 0.12 (including Esperanto localization) to the Market.

Ive downloaded it, thanks.

Quote from: axin on 30. April 2010, 10:02:47
Currently I don't have the JavaME source set up, but I guess Gert will have look as soon as he has a minute (@Gert...).

I have the sources up and running, so no need to publish a new J2ME version now, as people should be able to install straight from my test site (http://javabog.dk/filer/paroj/).
The problem was that Android apps are is not easily installable from other places than the Android market.

Quote from: axin on 30. April 2010, 10:02:47
Could you post some steps on how to support microemu and how to create the jnlp files? Or refer me to the relevant documentation?

See http://www.microemu.org/microemu-webstart/index.html

For JAD URL: http://wintermute.de/wap/extended/5ud0ku.jad
just point to:  http://microemu.org/webstart/wintermute.de/wap/extended/5ud0ku.jnlp
Replace bold part of URL with anything.

For example for http://javabog.dk/filer/paroj/eo-en/DictionaryForMIDs_eo-en_lernu.jad
I link to http://microemu.org/webstart/javabog.dk/filer/paroj/eo-en/DictionaryForMIDs_eo-en_lernu.jnlp

If you prefer running as applet instead of web start (see http://www.microemu.org/applet.html) there is a bug:
http://code.google.com/p/microemu/issues/detail?id=45&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary%20Reporter%20Stars
which requires you to change this code before it will work:

Index: JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/DictionaryForMIDs.java
===================================================================
--- JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/DictionaryForMIDs.java  (revision 271)
+++ JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/DictionaryForMIDs.java  (working copy)
@@ -57,7 +57,7 @@
                       
                        // check for supported MIDP version
                        String supportedMidpProfile = System.getProperty("microedition.profiles");
-                       if (supportedMidpProfile.indexOf("MIDP-2.") == -1) {
+                       if (supportedMidpProfile != null && supportedMidpProfile.indexOf("MIDP-2.") == -1) {
                                // if MIDP 2.0 is not supported, then the application will not run
                                utilObj.log("MIDP 2.0 not supported by the device.\n" +
                             applicationName + " will not run correctly !");


Wow - I am really very much impressed !!   :)  :)  :)

@Jacob, I gave you Developer status here in the forum, now you have access to the "Internal discussions" forum under "Admins & Developers". I will post more there soon.

With best regards,
Gert

Ok, now here my next few remarks:

-> The MicroEmu-app looks great !

Quote1) Updates to include Esperanto in the J2ME GUI:
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/UIDisplayTextContents.java
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/UIDisplayTextItems.java
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/DictionaryForMIDs.languages

Great, would you please just commit these updates to SVN (now you have the access rights).

Quote2) An update that will make DfM work with microemu.  See
http://code.google.com/p/microemu/issues/detail?id=44&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary%20Reporter%20Stars
The update should be deleted in 6 months (when a new version of microemu is released)

M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/lcdui_extension/DfMForm.java

That is DfMForm.java only, right ? Or also DictionaryForMIDs.java ?
Would you do me a favour: could you just add are remark to that line that you entered/modified (sort of "// introduced cause of MicroEmu; to be removed after xxx") ?

Quote3) This is most important:

A         DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java

Just a minor point: could you include your name (or 'identification') in the file header ?

And: could you update that web page:  http://dictionarymid.sourceforge.net/newdictNormationLang.html ? Well, just a question  ::)

Best greetings :)
Gert

Quote from: Gert on 01. May 2010, 01:07:14

Quote1) Updates to include Esperanto in the J2ME GUI:
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/UIDisplayTextContents.java
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/UIDisplayTextItems.java
M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/uidisplaytext/DictionaryForMIDs.languages

Great, would you please just commit these updates to SVN (now you have the access rights).

Here you are
http://dictionarymid.svn.sourceforge.net/viewvc/dictionarymid?view=rev&revision=279

Quote from: Gert on 01. May 2010, 01:07:14

Quote2) An update that will make DfM work with microemu.  See
http://code.google.com/p/microemu/issues/detail?id=44&colspec=ID%20Type%20Status%20Priority%20Milestone%20Owner%20Summary%20Reporter%20Stars
The update should be deleted in 6 months (when a new version of microemu is released)

M       JavaME/src/de/kugihan/dictionaryformids/hmi_java_me/lcdui_extension/DfMForm.java

That is DfMForm.java only, right ? Or also DictionaryForMIDs.java ?
Would you do me a favour: could you just add are remark to that line that you entered/modified (sort of "// introduced cause of MicroEmu; to be removed after xxx") ?

Done: http://dictionarymid.svn.sourceforge.net/viewvc/dictionarymid?view=rev&revision=278

Quote from: Gert on 01. May 2010, 01:07:14

Quote3) This is most important:

A         DictionaryForMIDs/src/de/kugihan/dictionaryformids/translation/normation/NormationEpo.java

Just a minor point: could you include your name (or 'identification') in the file header ?

And: could you update that web page:  http://dictionarymid.sourceforge.net/newdictNormationLang.html ? Well, just a question  ::)

[/quote]

Done :-)

Great !

Thanks a lot :)

Gert