Loading ...
Sorry, an error occurred while loading the content.
 

Re: win2k IME, multibyte/width characters, etc.

Expand Messages
  • Glenn F. Maynard
    ... Which makes it rather difficult to set up ... I don t know the widths of a font at a given point size (height), and don t know the exact way this setting
    Message 1 of 54 , Sep 22, 2001
      On Sat, Sep 22, 2001 at 01:02:49PM +0200, Bram Moolenaar wrote:
      > > set guifontwide=* doesn't do anything. (I don't think guifontset=* does,
      > > either.)
      >
      > No, this is not supported.

      Which makes it rather difficult to set up ... I don't know the widths of
      a font at a given point size (height), and don't know the exact way this
      setting is used. (It also seems to assume fonts are constant-width, and
      any Japanese font won't be; it'll have double-width characters for
      Japanese characters only, but--afaik--every Japanese font also contains
      single-width English characters.) The font selector appears to restrict
      itself to fixed-width fonts, but "fixed-width" fonts in Windows can
      still have double-width characters. So, I'm not sure if Vim is figuring
      this stuff out; if it's not, it'd probably explain why it thinks characters
      aren't supported and starts dumping hex.

      Anyhow, I'm going to concentrate on the IME and terminal stuff, since it's
      more concrete to me (I've done it before); I'll come back to this later.

      > How can your 'guifont' be "vga" if it doesn't work?

      I was hoping you'd know. It seems to work in _gvimrc, but not as a
      regular command.

      > Strange indeed. I hope someone can fix this. Or delete the
      > GetResultStr() function, if it's really not used.

      *sigh* Not encouraging. :) I'll snoop around the code (which I'm not
      familiar with) and let you know if I come up with anything (such as
      a real WM_IME_COMPOSITION handler, unless there's one hidden away
      somewhere that I'm missing.)

      > If you switch 'encoding', Vim may have the wrong bytes in the buffer.
      > What might happen here is that Vim first is in 8-bit mode, and sends
      > those bytes to the system functions as if they were normal printable
      > characters. Somehow the system recognizes the language and displays the
      > characters as if they were multi-byte characters, even though Vim
      > expected one character displayed for each byte.
      >
      > Did you try setting 'encoding' to "utf-8" and then pasting the text?

      The same thing happens.

      > Might be a Putty problem. Vim works fine in a utf-8 xterm. You could
      > try doing the same movements in an utf-8 xterm and check it works OK
      > there.

      It's hard to tell exactly what's being sent; I'll have to set up a telnet
      (can't snoop my own ssh) and see what's being sent, and if I can narrow it
      down to a problem in Putty or in Vim.

      --
      Glenn Maynard
    • Muraoka Taro
      Sorry, I forgot to attach that two files. ... diff -cr src.orig/mbyte.c src/mbyte.c ... *************** ... if (enc_utf8 && !option_was_set((char_u *) fencs ))
      Message 54 of 54 , Oct 25, 2001
        Sorry, I forgot to attach that two files.

        > I had found two problems about Glenn's latest patch for IME.
        >
        > 1. In mbyte.c FEAT_WINDOWS is used for instead of WIN32.
        > It cause touble when compiling on non-windows platform.
        > Check attached ime_fix1.diff.
        > 2. I got report of strange behavior from an user of old IME (it is called SKK).
        > I had checked this problem.
        > SKK send WM_IME_COMPOSITION message when 'k' typed
        > without any composition string.
        > It cause unexpected two white space insertion.
        > I had fixed this.
        > Please check _OnImeComposition() and GetResultStr() in ime_fix2.diff.
        >
        > Regards.
        > ----
        > Muraoka Taro <koron@...>
        >

        ----ime_fix1.diff
        diff -cr src.orig/mbyte.c src/mbyte.c
        *** src.orig/mbyte.c Sun Oct 21 20:30:38 2001
        --- src/mbyte.c Sun Oct 21 20:30:33 2001
        ***************
        *** 572,577 ****
        --- 572,591 ----
        if (enc_utf8 && !option_was_set((char_u *)"fencs"))
        set_string_option_direct((char_u *)"fencs", -1,
        (char_u *)"ucs-bom,utf-8,latin1", OPT_FREE);
        + #ifdef FEAT_MBYTE_IME
        + # ifdef USE_ICONV
        + ime_conv.vc_fd = (iconv_t)-1;
        + # endif
        + convert_setup(&ime_conv, "ucs-2", p_enc);
        + #endif
        +
        + #ifdef FEAT_MBYTE_IME
        + # ifdef USE_ICONV
        + ime_conv_cp.vc_fd = (iconv_t)-1;
        + # endif
        + ime_conv_cp.vc_type = CONV_CODEPAGE;
        + ime_conv_cp.vc_factor = 2; /* we don't really know anything about the codepage */
        + #endif

        #ifdef HAVE_BIND_TEXTDOMAIN_CODESET
        /* GNU gettext 0.10.37 supports this feature: set the codeset used for
        ***************
        *** 3539,3566 ****
        add_to_input_buf(str, 3);
        }

        - /*
        - * Add "str[len]" to the input buffer while escaping CSI bytes.
        - */
        - static void
        - add_to_input_buf_csi(char_u *str, int len)
        - {
        - int i;
        - char_u buf[2];
        -
        - for (i = 0; i < len; ++i)
        - {
        - add_to_input_buf(str + i, 1);
        - if (str[i] == CSI)
        - {
        - /* Turn CSI into K_CSI. */
        - buf[0] = KS_EXTRA;
        - buf[1] = (int)KE_CSI;
        - add_to_input_buf(buf, 2);
        - }
        - }
        - }
        -
        static GSList *key_press_event_queue = NULL;
        static int preedit_buf_len = 0;
        static gboolean processing_queued_event = FALSE;
        --- 3553,3558 ----
        ***************
        *** 3897,3902 ****
        --- 3889,3908 ----
        /* Internal utf-8 -> latin1 conversion. */
        vcp->vc_type = CONV_TO_LATIN1;
        }
        + #ifdef WIN32
        + /* Win32-specific UTF-16 -> DBCS conversion, for the IME,
        + * so we don't need iconv ... */
        + else if((from_prop & ENC_UNICODE) && (from_prop & ENC_2BYTE) && (to_prop & ENC_DBCS))
        + {
        + vcp->vc_type = CONV_DBCS;
        + vcp->vc_factor = 2; /* up to twice as long */
        + vcp->dbcs = atoi(to + 2);
        + } else if((from_prop & ENC_UNICODE) && (from_prop & ENC_2BYTE) && (to_prop & ENC_UNICODE)) {
        + vcp->vc_type = CONV_DBCS;
        + vcp->vc_factor = 2; /* up to twice as long */
        + vcp->dbcs = CP_UTF8;
        + }
        + #endif
        # ifdef USE_ICONV
        else
        {
        ***************
        *** 4023,4029 ****
        --- 4029,4077 ----
        retval = iconv_string(vcp->vc_fd, ptr, len);
        if (retval != NULL && lenp != NULL)
        *lenp = (int)STRLEN(retval);
        + break;
        # endif
        + # ifdef WIN32
        + case CONV_DBCS: /* UTF-16 -> dbcs or UTF8 */
        + {
        + int retlen;
        + if (!lenp)
        + len /= sizeof(unsigned short);
        +
        + retlen = WideCharToMultiByte(vcp->dbcs, 0,
        + (const unsigned short *)ptr, len, 0, 0, 0, 0);
        +
        + retval = alloc(retlen + 1);
        + if (retval == NULL)
        + break;
        +
        + WideCharToMultiByte(vcp->dbcs, 0,
        + (const unsigned short *) ptr, len, retval, retlen, 0, 0);
        +
        + retval[retlen] = NUL;
        + if (lenp != NULL)
        + *lenp = retlen;
        + break;
        + }
        + case CONV_CODEPAGE: /* current codepage -> ucs-2 */
        + {
        + int retlen;
        + retlen = MultiByteToWideChar(GetACP(), 0, ptr, len,
        + 0, 0);
        +
        + retval = alloc(sizeof(unsigned short) * retlen);
        + if (retval == NULL)
        + break;
        +
        + MultiByteToWideChar(GetACP(), 0, ptr, len,
        + (unsigned short *) retval, retlen);
        +
        + if (lenp != NULL)
        + *lenp = retlen * sizeof(unsigned short); /* number of shorts -> buffer size */
        + }
        +
        + # endif
        +
        }

        return retval;


        ----ime_fix2.diff
        diff -cr src.orig/gui_w32.c src/gui_w32.c
        *** src.orig/gui_w32.c Wed Sep 19 21:43:49 2001
        --- src/gui_w32.c Tue Oct 2 17:31:11 2001
        ***************
        *** 210,215 ****
        --- 210,219 ----
        static int get_toolbar_bitmap(vimmenu_T *menu);
        #endif

        + #ifdef FEAT_MBYTE_IME
        + static LRESULT _OnImeComposition(HWND hwnd, WPARAM dbcs, LPARAM param);
        + static char_u *GetResultStr(HWND hwnd, int GCS);
        + #endif
        #if defined(FEAT_MBYTE_IME) && defined(DYNAMIC_IME)
        # ifdef NOIME
        typedef struct tagCOMPOSITIONFORM {
        ***************
        *** 221,227 ****
        # endif

        HINSTANCE hLibImm = NULL;
        ! LONG (WINAPI *pImmGetCompositionString)(HIMC, DWORD, LPVOID, DWORD);
        HIMC (WINAPI *pImmGetContext)(HWND);
        BOOL (WINAPI *pImmReleaseContext)(HWND, HIMC);
        BOOL (WINAPI *pImmGetOpenStatus)(HIMC);
        --- 225,232 ----
        # endif

        HINSTANCE hLibImm = NULL;
        ! LONG (WINAPI *pImmGetCompositionStringA)(HIMC, DWORD, LPVOID, DWORD);
        ! LONG (WINAPI *pImmGetCompositionStringW)(HIMC, DWORD, LPVOID, DWORD);
        HIMC (WINAPI *pImmGetContext)(HWND);
        BOOL (WINAPI *pImmReleaseContext)(HWND, HIMC);
        BOOL (WINAPI *pImmGetOpenStatus)(HIMC);
        ***************
        *** 232,238 ****
        BOOL (WINAPI *pImmGetConversionStatus)(HIMC, LPDWORD, LPDWORD);
        static void dyn_imm_load(void);
        #else
        ! # define pImmGetCompositionString ImmGetCompositionStringA
        # define pImmGetContext ImmGetContext
        # define pImmReleaseContext ImmReleaseContext
        # define pImmGetOpenStatus ImmGetOpenStatus
        --- 237,244 ----
        BOOL (WINAPI *pImmGetConversionStatus)(HIMC, LPDWORD, LPDWORD);
        static void dyn_imm_load(void);
        #else
        ! # define pImmGetCompositionStringA ImmGetCompositionStringA
        ! # define pImmGetCompositionStringW ImmGetCompositionStringW
        # define pImmGetContext ImmGetContext
        # define pImmReleaseContext ImmReleaseContext
        # define pImmGetOpenStatus ImmGetOpenStatus
        ***************
        *** 790,795 ****
        --- 841,852 ----
        case WM_IME_NOTIFY:
        if (!_OnImeNotify(hwnd, (DWORD)wParam, (DWORD)lParam))
        return DefWindowProc(hwnd, uMsg, wParam, lParam);
        + break;
        +
        + case WM_IME_COMPOSITION:
        + if (!_OnImeComposition(hwnd, wParam, lParam))
        + return DefWindowProc(hwnd, uMsg, wParam, lParam);
        + break;
        #endif

        default:
        ***************
        *** 1292,1298 ****
        #include <ime.h>
        #include <imm.h>

        - static char lpCompStr[100]; // Pointer to composition str.
        static BOOL bInComposition=FALSE;

        /*
        --- 1352,1357 ----
        ***************
        *** 1356,1453 ****
        }
        }
        gui_update_cursor(TRUE, FALSE);
        ! lResult = 1;
        break;
        }
        pImmReleaseContext(hWnd, hImc);
        return lResult;
        }

        ! /* get composition string from WIN_IME */
        ! static void
        ! GetCompositionStr(HWND hwnd, LPARAM CompFlag)
        {
        ! DWORD dwBufLen; // Stogare for len. of composition str
        ! HIMC hIMC; // Input context handle.
        !
        ! // If fail to get input context handle then do nothing.
        ! // Applications should call ImmGetContext API to get
        ! // input context handle.

        ! if (!pImmGetContext || !(hIMC = pImmGetContext(hwnd)))
        ! return;

        ! // Determines how much memory space to store the composition string.
        ! // Applications should call ImmGetCompositionString with
        ! // GCS_COMPSTR flag on, buffer length zero, to get the bullfer
        ! // length.

        ! if ((dwBufLen = pImmGetCompositionString(hIMC, GCS_COMPSTR,
        ! (void FAR*)NULL, 0l)) < 0)
        ! goto exit2;

        ! if (dwBufLen > 99)
        ! goto exit2;

        ! // Reads in the composition string.
        ! if ( dwBufLen != 0 )
        {
        ! pImmGetCompositionString(hIMC, GCS_COMPSTR, lpCompStr, dwBufLen);
        ! lpCompStr[dwBufLen] = 0;
        }
        else
        {
        ! strcpy(lpCompStr, " ");
        ! dwBufLen = 2;
        ! }
        !
        ! // Display new composition chars.
        ! DisplayCompStringOpaque(lpCompStr, dwBufLen);


        ! exit2:
        ! pImmReleaseContext(hwnd, hIMC);
        }

        !
        ! // void GetResultStr()
        ! //
        ! // This handles WM_IME_COMPOSITION with GCS_RESULTSTR flag on.
        ! //
        ! // get complete composition string
        !
        ! static void
        ! GetResultStr(HWND hwnd)
        {
        ! DWORD dwBufLen; // Storage for length of result str.
        ! HIMC hIMC; // Input context handle.

        - // If fail to get input context handle then do nothing.
        if (!pImmGetContext || !(hIMC = pImmGetContext(hwnd)))
        ! return;
        !
        ! // Determines how much memory space to store the result string.
        ! // Applications should call ImmGetCompositionString with
        ! // GCS_RESULTSTR flag on, buffer length zero, to get the bullfer
        ! // length.
        ! if ((dwBufLen = pImmGetCompositionString(hIMC, GCS_RESULTSTR,
        ! (void FAR *)NULL, (DWORD) 0)) <= 0)
        ! goto exit2;
        !
        ! if (dwBufLen > 99)
        ! goto exit2;
        !
        ! // Reads in the result string.
        ! pImmGetCompositionString(hIMC, GCS_RESULTSTR, lpCompStr, dwBufLen);

        ! // Displays the result string.
        ! DisplayCompStringOpaque(lpCompStr, dwBufLen);

        ! exit2:
        pImmReleaseContext(hwnd, hIMC);
        }

        ! static char *
        ImeGetTempComposition(void)
        {
        if (bInComposition == TRUE)
        --- 1415,1523 ----
        }
        }
        gui_update_cursor(TRUE, FALSE);
        ! lResult = 0;
        break;
        }
        pImmReleaseContext(hWnd, hImc);
        return lResult;
        }

        ! static LRESULT
        ! _OnImeComposition(HWND hwnd, WPARAM dbcs, LPARAM param)
        {
        ! char_u *ret;
        ! if ((param & GCS_RESULTSTR) == 0) /* Composition unfinished. */
        ! return 0;

        ! if (ret = GetResultStr(hwnd, GCS_RESULTSTR))
        ! {
        ! add_to_input_buf_csi(ret, strlen(ret));
        ! vim_free(ret);
        ! return 1;
        ! }
        ! else
        ! return 0;
        ! }

        ! /* get the currnet composition string, in UCS-2; len is the number of
        ! * Unicode characters */
        ! unsigned short *
        ! GetCompositionString_inUCS2(HIMC hIMC, DWORD GCS, int *len)
        ! {
        ! LONG ret;
        ! unsigned short *wbuf = NULL;

        ! if (!pImmGetContext)
        ! return NULL; /* no imm32.dll */

        ! /* Try Unicode; this'll always work on NT regardless of codepage. */
        ! ret = pImmGetCompositionStringW(hIMC, GCS, NULL, 0);
        ! if (ret == 0)
        ! return NULL; /* empty */

        ! if (ret > 0)
        {
        ! wbuf = (unsigned short *) alloc(ret * sizeof(unsigned short));
        ! if(!wbuf) return NULL;
        !
        ! pImmGetCompositionStringW(hIMC, GCS, wbuf, ret);
        ! *len = ret / sizeof(unsigned short); /* char -> wchar */
        ! return wbuf;
        }
        +
        + /* ret < 0; we got an error, so try the ANSI version. This'll work
        + * on 9x/ME, but only if the codepage happens to be set to whatever
        + * we're inputting. */
        +
        + ret = pImmGetCompositionStringA(hIMC, GCS, NULL, 0);
        + if (ret <= 0)
        + return NULL; /* empty or error */
        else
        {
        ! char_u *buf;

        + buf = (char_u *) alloc(ret);
        + if (!buf)
        + return NULL;
        + pImmGetCompositionStringA(hIMC, GCS, buf, ret);

        ! /* convert from codepage to UCS-2 */
        ! wbuf = (unsigned short *)string_convert(&ime_conv_cp, buf, &ret);
        ! vim_free(buf);
        ! *len = ret / sizeof(unsigned short); /* char_u -> wchar */
        ! }
        ! return wbuf;
        }

        ! /*
        ! * void GetResultStr()
        ! *
        ! * This handles WM_IME_COMPOSITION with GCS_RESULTSTR flag on.
        ! * get complete composition string
        ! */
        ! static char_u *
        ! GetResultStr(HWND hwnd, int GCS)
        {
        ! DWORD dwBufLen; /* Stogare for len. of composition str. */
        ! HIMC hIMC; /* Input context handle. */
        ! unsigned short *buf = NULL;
        ! char *convbuf = NULL;

        if (!pImmGetContext || !(hIMC = pImmGetContext(hwnd)))
        ! return NULL;

        ! /* Reads in the composition string. */
        ! buf = GetCompositionString_inUCS2(hIMC, GCS, &dwBufLen);
        ! if (!buf)
        ! return NULL;

        ! convbuf = string_convert(&ime_conv, (unsigned char *) buf, &dwBufLen);
        pImmReleaseContext(hwnd, hIMC);
        + vim_free(buf);
        + return convbuf;
        }

        ! static char_u *
        ImeGetTempComposition(void)
        {
        if (bInComposition == TRUE)
        ***************
        *** 1459,1466 ****
        {
        pImmGetConversionStatus(hImc, &dwConvMode, &dwSentMode);
        pImmReleaseContext(s_hwnd, hImc);
        ! if ((dwConvMode & IME_CMODE_NATIVE))
        ! return lpCompStr;
        }
        }
        return NULL;
        --- 1529,1537 ----
        {
        pImmGetConversionStatus(hImc, &dwConvMode, &dwSentMode);
        pImmReleaseContext(s_hwnd, hImc);
        ! if ((dwConvMode & IME_CMODE_NATIVE)) {
        ! return GetResultStr(s_hwnd, GCS_COMPSTR);
        ! }
        }
        }
        return NULL;
        ***************
        *** 1668,1674 ****
        HBRUSH hbr;
        RECT rc;
        #ifdef FEAT_MBYTE_IME
        ! char *szComp;
        #endif

        if (!(flags & DRAW_TRANSP))
        --- 1739,1745 ----
        HBRUSH hbr;
        RECT rc;
        #ifdef FEAT_MBYTE_IME
        ! char_u *szComp;
        #endif

        if (!(flags & DRAW_TRANSP))
        ***************
        *** 1763,1772 ****
        {
        /* draw an incomplete composition character (korean) */
        if (len == 1 && blink_state == BLINK_ON
        ! && (szComp = ImeGetTempComposition()) != NULL) // hangul
        HanExtTextOut(s_hdc, TEXT_X(col), TEXT_Y(row),
        foptions, pcliprect, szComp, 2, padding, TRUE);
        ! else
        HanExtTextOut(s_hdc, TEXT_X(col), TEXT_Y(row),
        foptions, pcliprect, (char *)text, len, padding, FALSE);
        }
        --- 1834,1844 ----
        {
        /* draw an incomplete composition character (korean) */
        if (len == 1 && blink_state == BLINK_ON
        ! && (szComp = ImeGetTempComposition()) != NULL) { // hangul
        HanExtTextOut(s_hdc, TEXT_X(col), TEXT_Y(row),
        foptions, pcliprect, szComp, 2, padding, TRUE);
        ! vim_free(szComp);
        ! } else
        HanExtTextOut(s_hdc, TEXT_X(col), TEXT_Y(row),
        foptions, pcliprect, (char *)text, len, padding, FALSE);
        }
        ***************
        *** 3334,3342 ****
        hLibImm = LoadLibrary("imm32.dll");
        if (hLibImm == NULL)
        return;
        ! if ((*((FARPROC*)&pImmGetCompositionString)
        = GetProcAddress(hLibImm, "ImmGetCompositionStringA")))
        nImmFunc++;
        if ((*((FARPROC*)&pImmGetContext)
        = GetProcAddress(hLibImm, "ImmGetContext")))
        nImmFunc++;
        --- 3406,3417 ----
        hLibImm = LoadLibrary("imm32.dll");
        if (hLibImm == NULL)
        return;
        ! if ((*((FARPROC*)&pImmGetCompositionStringA)
        = GetProcAddress(hLibImm, "ImmGetCompositionStringA")))
        nImmFunc++;
        + if ((*((FARPROC*)&pImmGetCompositionStringW)
        + = GetProcAddress(hLibImm, "ImmGetCompositionStringW")))
        + nImmFunc++;
        if ((*((FARPROC*)&pImmGetContext)
        = GetProcAddress(hLibImm, "ImmGetContext")))
        nImmFunc++;
        ***************
        *** 3362,3368 ****
        = GetProcAddress(hLibImm, "ImmGetConversionStatus")))
        nImmFunc++;

        ! if (nImmFunc != 9)
        {
        FreeLibrary(hLibImm);
        hLibImm = NULL;
        --- 3437,3443 ----
        = GetProcAddress(hLibImm, "ImmGetConversionStatus")))
        nImmFunc++;

        ! if (nImmFunc != 10)
        {
        FreeLibrary(hLibImm);
        hLibImm = NULL;
      Your message has been successfully submitted and would be delivered to recipients shortly.