Change of Translation Layer interface: support for multiple dictionaries

[Gert]

Colleagues,

for years there had been no change of the interface to the Translation Layer - now there is one ! For the support for multiple dictionaries.

Until now, as a restriction, the Translation Layer only did support one dictionary at a time; this restriction came from the fact that information about the dictionary was stored in static fields of class DictionaryDataFile. I am currently updating the Translation Layer to work with several instances of DictionaryDataFile. This results in a few interface changes:

New in class TranslationExecution:
static DictionaryDataFile loadDictionary(DfMInputStreamAccess dictionaryDataFileISAccess);
static void               unloadDictionary(DictionaryDataFile );


The constructor of class TranslationParameters is extended with new first parameter dictionary:
TranslationParameters(DictionaryDataFile dictionary,
                      String             toBeTranslatedWordTextInput,
                      boolean[]          inputLanguages,
                      boolean[]          outputLanguages,
                      boolean            executeInBackground,
                      int                maxHits,
                      int                durationForCancelSearch)


TranslationResult.dictionary does indicate from which dictionary the translation was retrieved.


Removed will be the following:

Removed in class DictionaryDataFile:
initValues(boolean)

Removed class FileAccessHandler (complete class)



Changes required in the HMI Layer: [/font]

   
• the HMI Layer will have to use the return value of TranslationExecution.loadDictionary instead of the global DictionaryDataFile. This change can mostly be done by "search&replace" in the code.

Is this update ok ?? Or are there serious complaints ?


I myself plan to update the code for Java ME, Java SE,  DfM-Creator and WebApp mini-hmi. Achim, would that be possible that you update Android ? Actually, I think, the adaptation should be possible in one or two hours ... hope so 

I assume that this will be the last change of the interface to the Translation Layer for the next 15 years 

Best regards,
Gert

[Gert]

I already did adapt Java ME and Java SE to the new interface. I did not yet commit the changes to the SVN repository before I receive an 'OK'.

Here are a few hints for converting:

Old was:
FileAccessHandler.setDictionaryDataFileISAccess(dfmInputStreamObj);
DictionaryDataFile.initValues(false);

New is:
dictionary = TranslationExecution.loadDictionary(dfmInputStreamObj);


Updated is the access for the dictionary properties.

Old was (sample):
numLang = DictionaryDataFile.numberOfAvailableLanguages;

New is:
numLang = dictionary.numberOfAvailableLanguages;  // dictionary is returned from TranslationExecution.loadDictionary


And
Old was:

TranslationParameters translationParametersObj =
                  new TranslationParameters(addSearchCharacters(word),
                                          inputLanguages,
                                          outputLanguages,
                                          executeInBackground,
                                          DictionarySettings.getMaxHits(),
                                          DictionarySettings.getDurationForCancelSearch());


New is (new first parameter dictionary):
TranslationParameters translationParametersObj =
                  new TranslationParameters(dictionary,
                                          addSearchCharacters(word),
                                          inputLanguages,
                                          outputLanguages,
                                          executeInBackground,
                                          DictionarySettings.getMaxHits(),
                                          DictionarySettings.getDurationForCancelSearch());

Gert

[axin]

Hi Gert,

Great to see you already found some time to look into this.

I'm currently on the road with limited internet access so I wasn't able to check the changes you proposed in detail. When I was first thinking about the topic the first idea was to remove the static methods so that hmi could create multiple instances of the translation layer. Quickly looking at your description I see you design for the translation layer to take care of multiple dictionaries itself, which should be fine, too. Where I'm not sure now is how multithreading is working now. Will there be multiple calls providing results? Or do they come in a batch once? Does it work in synchronous mode or only in asynchronous mode?
I'm back home on Monday, then I can check more in detail.

Cheers
Achim

[Gert]

Achim,

wow - quick reaction !!

When I was first thinking about the topic the first idea was to remove the static methods

I was doing that for the class DictionaryDataFile (removing the static fields/methods), so now there is one instance of DictionaryDataFile per dictionary.

Where I'm not sure now is how multithreading is working now. Will there be multiple calls providing results? Or do they come in a batch once? Does it work in synchronous mode or only in asynchronous mode?

You hit the interesting point ! I was thinking quite a bit about how to do this. Finally, I did favour the presented solution, because there is only rather limited impact on the interface.

Let me explain: multithreading works as it did before. That means that a call to TranslationExecution.executeTranslation with executeInBackground will start a new thread that executes the translation. If you should do another call to TranslationExecution.executeTranslation before the previous translation completed, then that would interrupt the previous translation and start a new translation thread (this can happen for example with incremental search, where the user extended an expression before the translation results of the previous translation were there).

In order to run a translation over several dictionaries, you will have to run them one after the other. Each time when newTranslationResult is called, then the next translation can be executed.

Optionally, we could think to have an additional TranslationExecution.executeTranslationBatch where a list of translation jobs is passed in. Hmmmm, then TranslationResult could not carry the field dictionary, cause a TranslationResult would have several SingleTranslation each with a field dictionary. Guess that would not be hard to implement.

Hmmmm, or for a list of translation jobs there would be a list of TranslationResult, each for a separate dictionary (then the field dictionary in TranslationResult would be ok). Hmmmm, what are your ideas on that ?

With best greetings,
Gert

[Gert]

Achim,

I am currently implementing in class TranslationExecution

   void executeTranslationBatch(TranslationParametersBatch translationParametersBatchObj)

Basically a TranslationParametersBatch is a Vector with TranslationParameters elements.

This will allow to have one call to TranslationExecution and get the translation results for several dictionaries.

I plan to implement an own thread per dictionary translation. E.g. 5 dictionaries -> 5 threads running in parallel (I will have to rework threading and thread synchronization in any case, cause I plan support for Javascript web workers).

When the first dictionary translation is done, then a call to deletePreviousTranslationResult() will be done; subsequently newTranslationResult(resultOfTranslation) will be called for each dictionary.

Ok for you ? ... a simple posting with "OK" is enough 

Best regards,
Gert

[axin]

Hi Gert,

I had a look at the design you described. It sounds fine and should be relatively straight forward to adapt the Android HMI to use the changed API. But as always - the real problems will only emerge when really implementing it...

I'm currently not yet sure how the UI should change. Probably the user does not always want to search in all available dictionaries in all languages for translations. Maybe I'll add some language sets? So the user would not choose a dictionary+language but a set of dictionary+languages before starting translation.

Concerning executeTranslationBatch I guess we still have to work out some details. If the user has a lot of dictionaries with many languages or a device with limited resources spawning too many threads could cause problems. Furthermore, I quite like the idea of displaying all results belonging to one dictionary's language in a separate section in the result list. Therefore, I guess I'd prefer receiving one TranslationResult per dictionary per language...

For background information: Currently on Android I use a filterable ListAdapter which takes care of running the translation in a background thread. So I currently need to synchronously run translation in that thread. I remember having some problems with using executeInBackground=false, so in the end I used executeInBackground=true and some calls to Thread.wait() and Thread.notifyAll() to get synchronous behavior. Maybe I'll reconsider that when the new multi-threading translation API is available.

So this answer should be read as an "OK" from my side

Achim

[Gert]

So this answer should be read as an "OK" from my side

This makes me happy

I still need to spend some time on testing, then I will commit the updates to the SVN repository.

With best greetings,
Gert

[Gert]

I just commited my updates to SVN:

• DfM-Creator is adapted to the new interface, Karim, you simply need to update from SVN
• Achim, Android will need to be adapted
• Java ME is adapted
• Java SE is adapted
• WebApp: I will need to adapt the WebApp Translation Layer (currently the WebApp Java code will not compile), regenerate the Javascript code, and then adapt the mini_hmi - I will do this after updating the documentation on the web pages

I will update the documentation on the web pages as my next activity, Achim, with the information from the postings above, I believe it should be not so difficult to adapt to the changed interface.

Of course most of all, if there should be any problems or questions, then please ask !

Best regards,
Gert

[Gert]

I adapted the description at http://dictionarymid.sourceforge.net/development.html.

My next job will be the adaptation of WebApp Translation Layer.

Regards,
Gert

[Gert]

The WebApp Translation layer is adapted too now. I committed the updates to SVN.

This was quite an effort, also due to flaws in GWT.

The updated and extended documentation is here:
http://dictionarymid.sourceforge.net/WebApp/htdocs/WebAppDevelopment.html
http://dictionarymid.sourceforge.net/WebApp/htdocs/WebAppTranslationLayer.html

I still have to resolve a few "todo" points in the documentation.

I also adapted the mini_hmi to the new interface. I deliberately took one of those nice-looking versions from V.P. Guru for this. The result is here: http://svn.code.sf.net/p/dictionarymid/code/trunk/WebApp/Apps/mini_hmi/

I will work on deployment of the updated version next (means: adapting manifest files and putting everything on the Sourceforge server). And I will also update the documentation for this.

Note that right now the updates are not yet ready for testing. One of the reasons is that for the moment I postponed the implementation of web worker support in the Translation Layer and now there is a problem with reading binary data in Chrome (web apps can read binary data, but only in web workers).

Regards,
Gert

[axin]

Hi Gert,

I started migrating to the new API and using executeTranslationBatch() and already get good results, it's working great!

Now some questions arose and before digging into code I guess it's easier to ask you directly

- Does executeTranslationBatch() honor parameters.isExecuteInBackground or is this flag ignored and translation always runs in background?
- Is there a way to find out according to which TranslationParameters of the given batch the callback newTranslationResult(resultOfTranslation) is returning? I'd need this information to have a grouped view of the results, ie. output "English -> German" and the results, then "German -> English" and the corresponding results.

Cheers,
Achim

[Gert]

Hallo Achim,

- Does executeTranslationBatch() honor parameters.isExecuteInBackground or is this flag ignored and translation always runs in background?

Yes, this parameter is evaluated (for Java; not for Javascript); if set to false, no new thread is spawned. Uhm, at least that's what it is supposed to be ... is there a problem with that ?

- Is there a way to find out according to which TranslationParameters of the given batch the callback newTranslationResult(resultOfTranslation) is returning? I'd need this information to have a grouped view of the results, ie. output "English -> German" and the results, then "German -> English" and the corresponding results.

Hmmm, I think that is not possible. I will implement that !


Keep asking !!
Gert

[Gert]

Achim,

I added the member translationParametersObj to the class TranslationResult.

Does that solve your information need ... ?

Regards,
Gert

[axin]

Hi Gert,

sounds perfect, I just downloaded the updates from svn.

Thanks for that really fast enhancement
Achim

[axin]

- Does executeTranslationBatch() honor parameters.isExecuteInBackground or is this flag ignored and translation always runs in background?

Yes, this parameter is evaluated (for Java; not for Javascript); if set to false, no new thread is spawned. Uhm, at least that's what it is supposed to be ... is there a problem with that ?

No, there is no problem, I just wondered what would happen if some TranslationParameters in the batch have it set to true and some to false. But as I'm either setting all to execute in background OR all to execute synchronously this was just a theoretical question about the design

Cheers,
Achim