Messages

what code exactly needs be included into the DfM source ?

Pretty much all of the MainForm code that I posted seems to be needed:

The sleep subroutine to implement a forced delay between adding the items: It appears that it is a time sensitive problem.
The call of this when new items are added: Having this delay stops the (I suppose) failure of the SE properly registering the message calls (like 'paint') on the form's queue.
The forcing of background (threaded) execution of the dictionary search: Without this, the SE tries to update the form as items are added, but in a very broken way; a bit like a severe windows slowdown.
Moving the redisplay translation results call so that this is not done on the SE when the settings form is closed: Without this, the user can interact with the settings form while refreshAllTranslationResults(); is executing and since the added delay between item adding is in place, the process is much slowed too (depending on the number of results).

The sizeChanged() additions to the two CustomItem types classes are not essential, but probably also needed on a more general level just to make sure that when the display width is changed, the formatting is re-calculated so that eg: the VM adding a scrollbar does not cause the ends of lines to get hidden by the scrollbar.

well tested code

Put it this way, it works on my k800i 90-99%. I doubt there is a 100% fix because it is a problem with the VM on the phone. Without it, the custom items work about 0-2% of the time.
I have done all of the testing I have had the time and patience for; this was only with the one dictionary:- The English -> Chinese one and mostly with the test Midlet, which eventually ended up as the one I posted, though I think the method used in that of 'activating' the 'full functionality' of the CustomItems after they are added is not important to DFM: The 'sleep' works, and I think there would be too much to change (and too much potential for breaking things) to do this in DFM.

which does work on all devices

I can't know this because I only have a K800i to test on, so:
I have no idea if newer SE models have this bug.
I have no idea if older ones have it.
The problem appears to be time sensitive: Things break when an item is added when the previous one is still being processed by what appears to be a VM thread, so the delay may not be sufficient on older (read: maybe slower) SE phones if they have the same problem.

I can only appeal for any people who have other SE models who read this forum to try the fix on them!

My findings RE the SonyEricsson (KVM) bug:
http://dictionarymid.sourceforge.net/forum/index.php?topic=172.msg859#msg859

Great. now I can do something else

My findings on the SonyEricsson (KVM) Bug:
http://dictionarymid.sourceforge.net/forum/index.php?topic=172.msg859#msg859

Perhaps.
http://dictionarymid.sourceforge.net/forum/index.php?topic=172.msg859#msg859

The test MIDlet

Here the idea evolved a little.
Instead of needing a sleep between appends, we append our items in a 'lightweight' mode.
Then when we have done them all, we activate their simulated heavy overhead paint and formatting tasks
(their last formatting request invoked by sizeChanged is queued)

Code Select Expand


package TestForm;

import javax.microedition.midlet.*;
import javax.microedition.lcdui.*;

/**
* @author Tom Rowland
* a workaround for the SonyEricsson CustomItem on forms bug
*
* we append several CustomItems to our form that will do little
* until we 'activate' them.
* Once activated, wait(delay) is used to simulate overhead in their
* paint and reformatting methods
*
*/
public class TestMIDlet extends MIDlet implements CommandListener {

    static final int maxHits = 60;
    //
    private static final Command exitCommand = new Command("Exit", Command.EXIT, 0);
    private static final Command goCommand = new Command("GO", Command.OK, 0);
    private Form form;
    private StringItem stringItem;
    private ChoiceGroup checkBoxes;
    private String[] choices = {"Choice 1", "Choice 2", "Choice 3"};

    public TestMIDlet() {
        super();
        form = new Form("Test");
        form.addCommand(exitCommand);
        form.addCommand(goCommand);
        form.setCommandListener(this);
        initForm();
    }

    public void commandAction(Command command, Displayable displayable) {

        if (command == exitCommand) {
            notifyDestroyed();
        } else {
            doHits();
        }

    }

    void initForm() {
        form.deleteAll();
        stringItem = new StringItem(null, "Ready");
        form.append(stringItem);
        checkBoxes = new ChoiceGroup(null, ChoiceGroup.MULTIPLE, choices, null);
        form.append(checkBoxes);
    }

    void doHits() {
        initForm();
        //it appears that we MUST append CustomItems in a thread on
        //the SonyEricsson
        Thread t = new Thread() {

            public void run() {
                for (int count = 1; count <= maxHits; ++count) {
                    stringItem.setText("Count: " + Integer.toString(count));
                    synchronized (this) {
                        try {
                            form.append(new MyCustomItem("TextField " + Integer.toString(count), form.getWidth()));
                        //wait(3000L); no overhead between appends
                        } catch (Exception e) {
                        }
                    }
                }
                activateHits();
            }
        };
        t.start();
    }
    //after all of our CustomItems have been added,
    //turn on their simulated slow formatting and paint
    //routines

    private void activateHits() {
        //
        for (int i = 0; i < form.size(); ++i) {
            MyCustomItem item;
            if (form.get(i) instanceof MyCustomItem) {
                item = (MyCustomItem) form.get(i);
                item.activate();
            }
        }
    }

    public void startApp() {
        if (Display.getDisplay(this).getCurrent() != form) {
            Display.getDisplay(this).setCurrent(form);
        }
    }

    protected void pauseApp() {
    }

    protected void destroyApp(boolean unconditional) {
    }

    //a lightweight that turns heavyweight
    //a lightweight during the append to our form
    //a heavyweight after 'activation' (using waits^^)
    class MyCustomItem extends CustomItem {

        String label;
        int width, height, paints, formats;
        boolean needsFormatting, lightweightMode;

        MyCustomItem(String label, int screenWidth) {
            super(null);
            this.label = label;
            //lightweight mode
            lightweightMode = true;
            //set up our size
            reformat(screenWidth);
        }

        //called when 1st shown
        //called when a scrollbar is added too
        //ques a format if not 'lightweightMode' for our SE
        protected void sizeChanged(int w, int h) {
            if (w != width || h != height) {
                if (lightweightMode) {
                    //remember new width and set-up to
                    //call our reformat routine in the next paint
                    needsFormatting = true;
                    width = w;
                    return;
                }
                if (w != width) {
                    reformat(w);
                }
                /*
                It should be noted that invalidate can result
                in another call to sizeChanged according to the
                J2ME docs.
                We could have a problem if for some unknown reason
                the KVM continually refuses to recognize our desired
                contents height, but it seems to be ok.
                 */
                if (h != height) {
                    invalidate();
                }
            }
        }

        //Simulates a heavy overhead reformat
        private void reformat(int width) {
            ++formats;
            this.width = width;
            height = Font.getDefaultFont().getHeight() << 1;
            synchronized (this) {
                try {
                    //simulated processing is not too
                    //heavy for this test
                    wait(250L);
                } catch (Exception e) {
                }
            }
            needsFormatting = false;
            repaint();
        }

        protected int getMinContentWidth() {
            return width;
        }

        protected int getMinContentHeight() {
            return height;
        }

        protected int getPrefContentWidth(int height) {
            return this.width;
        }

        protected int getPrefContentHeight(int width) {
            return this.height;
        }

        //until we 'activate' our item, the 'heavy' formatting
        //and paint tasks are lightweight
        public void activate() {
            lightweightMode = false;
            repaint();
        }

        //Simulates a (very) heavy paint task
        //but ony draw the background until we are 'lightweightMode'
        //for the SonyEricsson. ie. after all our items have been
        //appended/insertes onto the form.
        protected void paint(Graphics g, int w, int h) {
            //lol, really we cannot do ANYTHING in paint
            //when we append many items quickly
            //well, we may get away with a 1 line box!
            //but often not 2 lines!!
            if (lightweightMode) {
                return;
            }
            if (needsFormatting) {
                reformat(width);
            }
            ++paints;
            synchronized (this) {
                try {
                    //simulated processing is not too
                    //heavy for this test
                    wait(50L);
                } catch (Exception e) {
                }
            }
            g.setColor(0x008080);
            g.fillRect(0, 0, width, height);
            g.setColor(0x0);
            g.setFont(Font.getDefaultFont());
            g.drawString(label, 0, 0, Graphics.TOP | Graphics.LEFT);
            g.drawString("Paints: " + Integer.toString(paints) +
                    "Formats: " + Integer.toString(formats),
                    0, Font.getDefaultFont().getHeight(), Graphics.TOP | Graphics.LEFT);
        }
    }
}


Sometimes while our items are being 'activated' we will have to scroll them off screen and back on so that they refresh.
This may be problematic when you have not used more height than the display, but also, the activation will take up less time, so the SE user may not have the problem.

I'm going back to the Canvas me

[In StringColourItem]

We can still get caught out with some incorrectly sized CustomItems on the SE; sometimes they will be set to the phone's screen height, so we need to call invalidate() in our items if sizeChanged is called with a height that is not what our Item is calculated to be during the formatting of the contents.

In StringColourItem

Code Select Expand


    protected void sizeChanged(int w, int h) {
        if (w != width || height != h) {
            setWidth(w);
            if (h != height) {
                invalidate();
            }
        }
    }


*This should be ok, but I think I should point out that the WTK docs do mention that invalidate() can result in a call to sizeChanged(). However I have not experienced a looping effect where invalidate() has not fixed the height and sizeChanged has been re-called.

In BitmapFontCanvas
This is a little more complicated (and I did not study the class much, so what I have done here to allow re-formatting if the width changes (usually when a scrollbar is added) may not be optimal for it.

Code Select Expand


    public BitmapFontCanvas(StringColourItemText input, String fontSize, int maxWidthPixels, boolean colouredMode) {
        super(null);
        this.stringItem = input;
        this.bitmapFontSize = fontSize;
        this.colouredMode = colouredMode;
        setWidth(maxWidthPixels);
    }

    private void setWidth(int newWidth) {
        if (maxWidthPixels != newWidth) {
            try {
                maxWidthPixels = newWidth;
                String fontDir;
                fontDir = "/dictionary/fonts/" + bitmapFontSize + "/";
                font = BitmapFont.getInstance(fontDir, fontDir + "font.bmf");
                font.loadFont();
                font.loadChars();
                viewer = font.getViewer(stringItem, maxWidthPixels, colouredMode);
                lineHeightPixels = font.getLineHeightPixels();
                totalHeightPixels = viewer.getLinesPainted() * lineHeightPixels;
            } catch (Exception e) {
            }
            repaint();
        }
    }
    protected void sizeChanged(int w, int h) {
        if (w != maxWidthPixels || h != totalHeightPixels) {
            setWidth(w);
            if (h != totalHeightPixels) {
                invalidate();
            }
        }
    }


[New method for our sleep after inserting a result]

Changes to MainForm.

New method for our sleep after inserting a result

Code Select Expand


    public void applySonyEricssonWorkaroundSleep() {
        if (sonyEricssonWorkaroundRequired) {
            long t = System.currentTimeMillis();
            synchronized (this) {
                try {
                    Thread.sleep(300L);
                } catch (Exception e) {
                    Util.getUtil().log("sonyEricssonWorkaround: " + e.toString() + " / " + e.getMessage());
                }
            }
        }
    }


Called from addTranslationItem(Item item)

Code Select Expand


     void addTranslationItem(Item item) {
        ++indexOfLastTranslationItem;
        insert(indexOfLastTranslationItem, item);
        /*
        This SHOULD fix problems with SonyEricsson adding of CustomItems to the form
        It MAY fix problems that some have reported with normal Items too
        (testing is needed)
         */
        applySonyEricssonWorkaroundSleep();
    }


Force execution in background in translateWord(....)

Code Select Expand


    public void translateWord(String word,
            boolean executeInBackground)
            throws DictionaryException {
        if (DictionarySettings.isDictionaryAvailable()) {
            // for JSR75 support this needs to be done in a separate thread
            if (DictionarySettings.isUseFileAccessJSR75()) {
                executeInBackground = true;
            } else
            // force execution in background for SonyEricsson
            if (sonyEricssonWorkaroundRequired) {
                executeInBackground = true;
            }

            ....


(while I'm at it, a wee bug fix to clear the last search results status in the subroutine that is just above that last snippet)

Code Select Expand


    public void translateToBeTranslatedWordTextField(boolean executeInBackground)
            throws DictionaryException {
        removeStartupDisplay();
        //clear last translation results status
        translationResultStatus.setLabel("");
        translateWord(toBeTranslatedWordTextField.getString(),
                executeInBackground);
    }


Reposition the if statement in updateMainFormItemsObj() so that we do not get left waiting on the settings form

Code Select Expand


        ....

        if (!sonyEricssonWorkaroundRequired) {
            // redisplay translation results
            refreshAllTranslationResults();

            //Causes an error on SonyEricsson devices when called before the midlet is started
            display.setCurrentItem(toBeTranslatedWordTextField);
        }
    }


There are several places on this forum where people have posted about their SonyEricsson not displaying the coloured text (StringColourItem) and bitmap fonts (BitmapFontCanvas). I am one of them (with a K800i).
Excuse the long post explaining what is behind this problem.

Over the last 3 days, I have spent some time looking into this problem and can now suggest a workaround, while not perfect is lots better. It is quite simple too.

So, What seems to be causing this problem?
Well.. ..With Forms (and I am not all that experienced in using them since I went the way of a custom UI on a canvas with my apps), the SE has a bit of a quirk to it. One that is not present on the Sony SDK emulators.

Normally, what you would expect when running some process is that the device would be in a state of 'lock', or completely unresponsive to user interaction, bar those that will interrupt the KVM (such as the method for forcing program closure). This is how it is with the WTK and SE emulators. The SE however is not really like this, for some strange reason they decided that the stuff that is behind their form implementation will still function. It would appear that you can still scroll the form and interact with the items on it. Items can be updated, and their new contents are shown. Items can be added and will get displayed on the form if say, you added one, then did some un-threaded processing and added another. With this last point, we are getting close to our problem.

Here is an example description of a MIDlet to demonstrate this quirk:

It consists of a Form, StringItem and ChoiceGroup with 3 check boxes.
There are 2 commands, one to 'exit', and one to 'go'.
Initially, the StringItem contains "Ready"

When 'go' is used, we loop from 1 to 10 and for each cycle of that loop:
We set the StringItem to "Count: "+loop cycle value.
We add a TextField containing "TextField "+loop cycle value.
We wait for 3 seconds.

With a 'Normal' phone, when 'go' is used, the phone is in a state of apparent 'lock' for about 30 seconds.
After this, the StringItem contains "Count: 10" and the 10 TextFields are displayed.

With the Sony, we see "Count: 1" ... "Count: 10".
We can change the check boxes.
We see The TextFields appear and can even edit them as they do!

So what has this got to do with the CustomItem problem?

We modify our MIDlet.
This time we append a CustomItem that displays our text instead.

Now what does the SE do?

The same thing DictionaryForMIDs does. Some items are painted, some not, several are the wrong size.

We have broken the Sony's background form processing; it would appear that when it was setting up one CustomItem on the form, another came along and it forgot about the first or something. This first item was (I presume) incompletely set-up on the form and now never gets things like its paint method called, even if you force it to request a repaint.

DictionaryForMIDs can execute the translation in the background, on a thread. Does this help? Yes. Things are fixed again for our test, but not in DictionaryForMIDs if we execute the translation in the background.

We have a lot of simulated overhead in our item appending loop, and very little overhead in our CustomItem compared to, say, BitmapFontCanvas. So its down to timing, I guess.

A Thread.sleep(long delay) will fix things in DictionaryForMIDs. I seem to be getting good success with about 300ms on my K800i and this is our simplest answer.

The delay causes some other problems:

• is it enough for other SE phones?
• is it enough for other dictionaries? (my test one is the English-Chinese one btw)

So this needs testing on other SonyEricsson phones.

Another problem also comes to light. DictionaryForMIDs needs to not call MainForm.refreshAllTranslationResults() from within MainForm.updateMainFormItemsObj() after accept is chosen on the settings form. The new delay on the inserting of items is going to cause a long delay as they are re-inserted. During that delay, the settings form can be 'toyed' with while the SE user waits

EDIT: I forgot; it appears that hiding the form is of no help.

Concerning the SE topic of your postings, we simply can split that to a new thread ("Split Topic", next to Quote, Modify, Remove). Hmmm, maybe you cannot, cause you do not have the right to do so ? If not, then tell me.

Yep I do not have the rights.

Besides, I assume that the formatting problems of your SE model only show up for the 'Coloured display' (which uses StringColourItem). If you switch off  'Coloured display', then I assume the output will not look nice (cause of the missing colouring), but the formatting will be ok, is this right ?

Yes the formatting is ok without. The fact is that really it is the bitmap font that I need to get working since I have no Chinese characters and it was this dictionary that bought me to your app.
The problems are related; it is the way the SE VM is handling customItem.

Before I split/move threads. Umm a potential indexOutOfBounds in StringColourItem or just paranoia?

Code Select Expand


    public void clearSelectedWord() {
            //if (currSelectItem >= 0 && currSelectItem < itemTextWrap.length)  //un-comment and add this!
            itemTextWrap[currSelectItem] = ClearSelectChar(itemTextWrap[currSelectItem]);
            selectedWord = null;
            currSelectItem = -1;
            repaint();
    }



The SE appears not to need this Font.stringwidth() workaround after all. Further tests have shown that I got fooled by the 'nothing showing up' problem being worse on some word translations than others, sometimes intermittent, and strangely often it is the same Items that will not get painted 

Items that are added to the form, but are not visible without scrolling are being set to an odd height with my SE; the display height. And this is why I thought there was a problem being caused by the undefined characters and Font.StringWidth() .

This seems to be fixable by forcing the VM to reset the layout using a call to invalidate(), though I need to do more work and testing to find out where is best. Right now putting it in showNotify() seems to work best.

Oh ... well,  if we only knew about the root of the problems in the SE Java implementation, then we might be able to find a working workaround.

I think that although some of the changes are going to be relevant to StringColourItem, it will take this thread more and more off topic to talk about the SE here. So I will move to a new one when I have more and post a link to that here.

The sizeChanged that I provided before may still be good to keep (testing is needed on other devices), when the VM adds a scrollbar, it would make sense to recalculate things, as a new width may cause the word wrapping to change and thus alter the height needed.

Tom

Besides, did you replace the plenty occurrences of Font.stringWidth with your workaround for testing ?

There are only about 4 calls in the class.
EDIT: missing thos outside of the formatting for the item size^^

Better not get too excited. Alas I have problems testing the Chinese version with ODD, so I cannot be sure if that 'apparent fix' is THE fix and I may have jumped the gun with it.
I will do some more ODD now I have set it up and try see for sure what was happening. I can use the English-Khmer ok with ODD and this has non-standard characters.

I deactivated sizeChanged in version 3.4 cause it was throwing Exceptions on some devices. Of course, if after some testing your proposal will work fine, I will be glad to incorporate it for 3.5 !

Really? what type of exception? The device would call this when the size changes ie. when the scrollbar is added as things become to tall for fitting on the screen all at once.

I am a bit confused here though; if the item size changes, you should call invalidate() to notify the device, but lol. the j2me docs say this can then result in a call to sizeChanged. Hmm. ok, maybe an

if(oldHeight!=newHeight etc.)
invalidate()

Would be needed to stop a nasty loop with that one.

That SE workaround(s) caused us a lot of nightmares; it was pure 'try and error'; and it seems that depending on the SE Java Platform and Java Platform revision number the bug shows up at a code location. I mean the bug within the SE Java implementation.

Well, maybe a simple Thread.yield() may have a good effect; we really would need to have someone with an affected device in order to test.

Agreed, an affected device is needed for the test.

In fact it is a problem to have on it.

Well, what is that problem ? Guess I forgot about this ...

hehe. it causes the items not to show; results in the reason for the workaround.

In my case, I think I managed to solve this by replacing the Font.stringWidth() calls in StringColourItem with a workaround;
It appears that on the device (in my case SE K800i), some of the 'non defined' characters are breaking the function.

see also: http://dictionarymid.sourceforge.net/forum/index.php?topic=145.0

My SE K800i has problems with the formatting with the English-Chinese dictionary.
I guessed that this was something to do with most of the characters being undefined, and breaking Font.stringWidth.
I tried a workaround to replace that function, which seems to be ok (well, a whole lot better on it in any case).

Code Select Expand


private int strWidthWorkaround(Font f, String s) {
 if (s == null || s.length() == 0)
     return 1;
 int w = 0;
 for (int i = 0; i < s.length(); ++i) {
  int wi = f.charWidth(s.charAt(i));
  if (wi <= 0 || wi >= 16)
      wi = f.charWidth('w');
  w += wi;
 }
 return w;
}


EDIT: the wi>=16 was just for the test I did. Perhaps use: wi>=f.charWidth('w') instead.

There is perhaps little point since a load of 'squares' in both the Chinese and Pinyin is hardly a translation worth using^^
Sadly there are, by the looks of it, similar problems with the BitmapFont. (I will need to set up on device debugging to get somewhere with this one, and dunno if I can be bothered).

I also think that for niceness, this (or similar) should be included:

Code Select Expand


 protected void sizeChanged(int w,int h) {
   setWidth(w);
 }


btw: The SE workaround in the main form is not needed on this device. In fact it is a problem to have on it. I wonder if a simple Thread.yield() (to allow a repaint) works for those that do need that workaround.