Loading ...
Sorry, an error occurred while loading the content.

Binary Representation

Expand Messages
  • Steve Amerige
    Some of the files I edit contain binary characters. For example, PostScript files frequently contain binary data. Presently, these data are hard to read. With
    Message 1 of 13 , May 4, 2001
    • 0 Attachment
      Some of the files I edit contain binary characters.
      For example, PostScript files frequently contain binary data.
      Presently, these data are hard to read. With file data ranging
      from 00 to FF (hex), and the normally viewable portion being
      from 20 to 7E, we have the not normally viewable portion being
      00 to 1F and 7F to FF.

      It would be nice to have a way of displaying codes in the non-
      viewable range as either:

      octal: \nnn
      hex: xx

      with some kind of optional highlighting to make it obvious that
      these are alternate representations of code values.

      This is not a request to see the entire file as hex or octal, just
      those codes that would not otherwise be self describing.

      Or..... by any chance is this doable today with Vim (windows) 5.7?

      Thanks,
      Steve Amerige
      Adobe Systems Incorporated
    • Matt Kunze
      ... try :he xxd for vim s answer to hex editing ga on a single character will print the ascii/decimal/hex value for that character
      Message 2 of 13 , May 4, 2001
      • 0 Attachment
        Steve Amerige wrote:

        >Some of the files I edit contain binary characters.
        >For example, PostScript files frequently contain binary data.
        >Presently, these data are hard to read. With file data ranging
        >from 00 to FF (hex), and the normally viewable portion being
        >from 20 to 7E, we have the not normally viewable portion being
        >00 to 1F and 7F to FF.
        >
        >It would be nice to have a way of displaying codes in the non-
        >viewable range as either:
        >
        > octal: \nnn
        > hex: xx
        >
        >with some kind of optional highlighting to make it obvious that
        >these are alternate representations of code values.
        >
        >This is not a request to see the entire file as hex or octal, just
        >those codes that would not otherwise be self describing.
        >
        >Or..... by any chance is this doable today with Vim (windows) 5.7?
        >
        >Thanks,
        >Steve Amerige
        >Adobe Systems Incorporated
        >
        try ':he xxd' for vim's answer to hex editing
        'ga' on a single character will print the ascii/decimal/hex value for
        that character
      • Jeff Walker
        You can also put that info in your status line with %b or %B. I do this, it takes away my need for anything else (usually). -- Jeff Walker
        Message 3 of 13 , May 4, 2001
        • 0 Attachment
          You can also put that info in your status line with %b or %B. I do this, it
          takes away my need for anything else (usually).

          --
          Jeff Walker MatchLogic, Inc.
          jwalker@... 7233 Church Ranch Blvd.
          Voice 1 (303) 222-2105 Westminster, CO 80021
          Fax 1 (303) 222-2001 www.matchlogic.com


          -----Original Message-----
          From: Matt Kunze [mailto:kunzem@...]
          Sent: Friday, May 04, 2001 4:38 PM
          To: Steve Amerige
          Cc: vim@...
          Subject: Re: Binary Representation


          Steve Amerige wrote:

          >Some of the files I edit contain binary characters.
          >For example, PostScript files frequently contain binary data.
          >Presently, these data are hard to read. With file data ranging
          >from 00 to FF (hex), and the normally viewable portion being
          >from 20 to 7E, we have the not normally viewable portion being
          >00 to 1F and 7F to FF.
          >
          >It would be nice to have a way of displaying codes in the non-
          >viewable range as either:
          >
          > octal: \nnn
          > hex: xx
          >
          >with some kind of optional highlighting to make it obvious that
          >these are alternate representations of code values.
          >
          >This is not a request to see the entire file as hex or octal, just
          >those codes that would not otherwise be self describing.
          >
          >Or..... by any chance is this doable today with Vim (windows) 5.7?
          >
          >Thanks,
          >Steve Amerige
          >Adobe Systems Incorporated
          >
          try ':he xxd' for vim's answer to hex editing
          'ga' on a single character will print the ascii/decimal/hex value for
          that character
        • Steve Amerige
          The suggestions below are but workarounds and are not entirely satisfactory: When binary data are present in a PostScript file, the highlighting gets put into
          Message 4 of 13 , May 4, 2001
          • 0 Attachment
            The suggestions below are but workarounds and are not entirely
            satisfactory:

            When binary data are present in a PostScript file, the highlighting
            gets put into a bad state because the highlighting has no way of
            knowing that the data are binary.

            The xxd solution is not acceptable at all because the file is mostly
            ASCII with some binary. I need to see the ASCII as ASCII, and
            the binary as an alternate representation using viewable ASCII.

            The %b or %B status line is not acceptable because it does not
            help when I'm looking for patterns in the text. We are accustomed
            to certain letterforms, and the binary representation that presently
            includes accented characters, symbols, and control characters are
            not among them. In addition to being difficult to distinguish one
            from another, some representations are ambiguous. Some code
            points map to the same symbol (e.g., 127, 129, 141, 143, 144 all
            map to a rectangle with Lucida Console as the font; 1 is indistinguishable
            from the sequence 94, 65; and so forth).

            What is needed is a general mechanism for glyph display, editing, and
            searching. Presently when I have to edit files with ASCII and binary
            data and need to see both, I'm compelled to use emacs--yikes!
            This is an area in which VIM could learn from emacs.

            Is there any chance of this happening?


            > You can also put that info in your status line with %b or %B. I do this, it
            > takes away my need for anything else (usually).
            >
            > try ':he xxd' for vim's answer to hex editing
            > 'ga' on a single character will print the ascii/decimal/hex value for
            > that character
          • Bram Moolenaar
            ... That s a problem of the syntax file. It can check if characters are printable or not and do something with that. ... In Vim 6.0 you can set the display
            Message 5 of 13 , May 5, 2001
            • 0 Attachment
              Steve Amerige wrote:

              > When binary data are present in a PostScript file, the highlighting
              > gets put into a bad state because the highlighting has no way of
              > knowing that the data are binary.

              That's a problem of the syntax file. It can check if characters are printable
              or not and do something with that.

              > The xxd solution is not acceptable at all because the file is mostly
              > ASCII with some binary. I need to see the ASCII as ASCII, and
              > the binary as an alternate representation using viewable ASCII.

              In Vim 6.0 you can set the 'display' option to "uhex" to get unprintable
              characters displayed as <xx>, where xx is the hexadecimal number of the
              character.

              > What is needed is a general mechanism for glyph display, editing, and
              > searching. Presently when I have to edit files with ASCII and binary
              > data and need to see both, I'm compelled to use emacs--yikes!
              > This is an area in which VIM could learn from emacs.

              Editing and search can be done with any characters, I don't see a problem
              here. Vim wasn't really made to edit binary data. You can do it, but it
              requires a bit more typing.

              --
              From "know your smileys":
              [:-) Frankenstein's monster

              /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
              ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
              \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
            • Thomas S. Urban
              ... ... Also, you can mess with the status line variable so that you always get the value of the character under the cursor: add %b for decimal value,
              Message 6 of 13 , May 5, 2001
              • 0 Attachment
                On Fri, May 04, 2001 at 04:38:25PM -0600, Matt Kunze wrote:
                > Steve Amerige wrote:
                >
                <snip>

                > >This is not a request to see the entire file as hex or octal, just
                > >those codes that would not otherwise be self describing.
                > >
                > try ':he xxd' for vim's answer to hex editing
                > 'ga' on a single character will print the ascii/decimal/hex value for
                > that character

                Also, you can mess with the status line variable so that you always get
                the value of the character under the cursor: add %b for decimal value,
                %B for hex value, see

                :he 'statusline'

                for more info.

                I'm not sure if this is in version 5.7 or not - I use the 6.0alphas.

                Scott

                --
                Suffocating together ... would create heroic camaraderie.
                -- Khan Noonian Singh, "Space Seed", stardate 3142.8
              • Steve Amerige
                ... It might be a bit difficult. As PostScript is a general programming language, the procedures that cause binary data to be read from within the file itself
                Message 7 of 13 , May 7, 2001
                • 0 Attachment
                  Bram Moolenaar wrote:

                  > Steve Amerige wrote:
                  >
                  > > When binary data are present in a PostScript file, the highlighting
                  > > gets put into a bad state because the highlighting has no way of
                  > > knowing that the data are binary.
                  >
                  > That's a problem of the syntax file. It can check if characters are printable
                  > or not and do something with that.

                  It might be a bit difficult. As PostScript is a general programming language,
                  the procedures that cause binary data to be read from within the file itself
                  cannot be enumerated. So, what's needed is some kind of smart switching:

                  if a binary (non-viewable) glyph is encountered, shift into a no-highlight mode.
                  Then when we can be "reasonably" sure we're not in a binary section (e.g.,
                  because a "known" PostScript token has been encountered), then shift back
                  into regular highlighting mode.

                  That's about the best heuristic I can think of, but I'm not fluent enough in VIM
                  constructs to be able to implement such a thing. I am very fluent in PostScript,
                  and I'll be glad to work with anyone who would like to improve our present
                  PostScript VIM file (postscr.vim). It would be nice if we could have a general
                  solution for this kind of thing, too.

                  > > The xxd solution is not acceptable at all because the file is mostly
                  > > ASCII with some binary. I need to see the ASCII as ASCII, and
                  > > the binary as an alternate representation using viewable ASCII.
                  >
                  > In Vim 6.0 you can set the 'display' option to "uhex" to get unprintable
                  > characters displayed as <xx>, where xx is the hexadecimal number of the
                  > character.
                  >
                  > > What is needed is a general mechanism for glyph display, editing, and
                  > > searching. Presently when I have to edit files with ASCII and binary
                  > > data and need to see both, I'm compelled to use emacs--yikes!
                  > > This is an area in which VIM could learn from emacs.
                  >
                  > Editing and search can be done with any characters, I don't see a problem
                  > here. Vim wasn't really made to edit binary data. You can do it, but it
                  > requires a bit more typing.

                  I'm glad to hear that VIM will support the displaying of unprintable characters.
                  How will one type them in for things such as searching or inserting them into
                  a document?

                  Thanks,
                  Steve Amerige
                  Adobe Systems Incorporated
                • Benji Fisher
                  ... [snip] ... I think only the display options are being improved. You can already (in vim 5.x) input unprintable characters. ... I do not use this much, but
                  Message 8 of 13 , May 7, 2001
                  • 0 Attachment
                    Steve Amerige wrote:
                    >
                    > Bram Moolenaar wrote:
                    >
                    > > Steve Amerige wrote:
                    [snip]
                    > > In Vim 6.0 you can set the 'display' option to "uhex" to get unprintable
                    > > characters displayed as <xx>, where xx is the hexadecimal number of the
                    > > character.
                    > >
                    > > > What is needed is a general mechanism for glyph display, editing, and
                    > > > searching. Presently when I have to edit files with ASCII and binary
                    > > > data and need to see both, I'm compelled to use emacs--yikes!
                    > > > This is an area in which VIM could learn from emacs.
                    > >
                    > > Editing and search can be done with any characters, I don't see a problem
                    > > here. Vim wasn't really made to edit binary data. You can do it, but it
                    > > requires a bit more typing.
                    >
                    > I'm glad to hear that VIM will support the displaying of unprintable characters.
                    > How will one type them in for things such as searching or inserting them into
                    > a document?

                    I think only the display options are being improved. You can
                    already (in vim 5.x) input unprintable characters.

                    :help i_CTRL-V_digit

                    I do not use this much, but I think it works the same way in Command
                    (Ex) mode.

                    HTH --Benji Fisher
                  • Steve Amerige
                    Hi all, ... Thanks. If VIM 6.0 will display binary characters in hex, it would be nice if one could enter characters in hex as well. The mechanism above
                    Message 9 of 13 , May 7, 2001
                    • 0 Attachment
                      Hi all,

                      Benji Fisher wrote:

                      > I think only the display options are being improved. You can
                      > already (in vim 5.x) input unprintable characters.
                      >
                      > :help i_CTRL-V_digit
                      >

                      Thanks. If VIM 6.0 will display binary characters in hex, it would be nice
                      if one could enter characters in hex as well. The mechanism above
                      doesn't allow one to enter decimal 10 (0x0a). Any input mechanism
                      should allow one to enter all possible codes (0x00 to 0xff).

                      As a tangent: can one edit Japanese or other non-Roman documents
                      with VIM as one might with editors such as mule (MUlti-Lingual Emacs)?
                      There are certainly related issues of input methods and display issues
                      here. I hope that VIM over time can embrace general-purpose input
                      methods and general-purpose display. I'm sure it's difficult, but it is
                      a worthy goal for some future incarnation of VIM.

                      Thanks again,
                      Steve Amerige
                      Adobe Systems Incorporated
                    • Benji Fisher
                      ... Vim is not a binary editor. It is a line-based editor. It needs some EOL character, and it uses 0x0a. You can enter a 0x0a by typing in Input mode.
                      Message 10 of 13 , May 7, 2001
                      • 0 Attachment
                        Steve Amerige wrote:
                        >
                        > Hi all,
                        >
                        > Benji Fisher wrote:
                        >
                        > > I think only the display options are being improved. You can
                        > > already (in vim 5.x) input unprintable characters.
                        > >
                        > > :help i_CTRL-V_digit
                        > >
                        >
                        > Thanks. If VIM 6.0 will display binary characters in hex, it would be nice
                        > if one could enter characters in hex as well. The mechanism above
                        > doesn't allow one to enter decimal 10 (0x0a). Any input mechanism
                        > should allow one to enter all possible codes (0x00 to 0xff).

                        Vim is not a binary editor. It is a line-based editor. It needs
                        some EOL character, and it uses 0x0a. You can enter a 0x0a by typing <CR>
                        in Input mode. (I think I have this straight.) Any other character can
                        be entered with <C-V>ddd. For example, <C-V>255 enters 0xff.

                        > As a tangent: can one edit Japanese or other non-Roman documents
                        > with VIM as one might with editors such as mule (MUlti-Lingual Emacs)?
                        > There are certainly related issues of input methods and display issues
                        > here. I hope that VIM over time can embrace general-purpose input
                        > methods and general-purpose display. I'm sure it's difficult, but it is
                        > a worthy goal for some future incarnation of VIM.

                        I do not use this myself, but start by reading

                        :help multibyte

                        HTH --Benji Fisher
                      • Bram Moolenaar
                        ... Well, you ll have to find some way to locate those binary areas, otherwise the syntax file is unable to find them. After all it s syntax highlighting, not
                        Message 11 of 13 , May 7, 2001
                        • 0 Attachment
                          Steve Amerige wrote:

                          > > > When binary data are present in a PostScript file, the highlighting
                          > > > gets put into a bad state because the highlighting has no way of
                          > > > knowing that the data are binary.
                          > >
                          > > That's a problem of the syntax file. It can check if characters are
                          > > printable or not and do something with that.
                          >
                          > It might be a bit difficult. As PostScript is a general programming language,
                          > the procedures that cause binary data to be read from within the file itself
                          > cannot be enumerated. So, what's needed is some kind of smart switching:

                          Well, you'll have to find some way to locate those binary areas, otherwise the
                          syntax file is unable to find them. After all it's syntax highlighting, not a
                          PostScript interpreter.

                          > if a binary (non-viewable) glyph is encountered, shift into a no-highlight
                          > mode. Then when we can be "reasonably" sure we're not in a binary section
                          > (e.g., because a "known" PostScript token has been encountered), then
                          > shift back into regular highlighting mode.
                          >
                          > That's about the best heuristic I can think of, but I'm not fluent enough in
                          > VIM constructs to be able to implement such a thing. I am very fluent in
                          > PostScript, and I'll be glad to work with anyone who would like to improve
                          > our present PostScript VIM file (postscr.vim). It would be nice if we could
                          > have a general solution for this kind of thing, too.

                          I don't think there is a generic solution. The parsing of the language is
                          different for every language. The easiest would be if there is some marker
                          before and after the part that should be handled differently.

                          > I'm glad to hear that VIM will support the displaying of unprintable
                          > characters. How will one type them in for things such as searching or
                          > inserting them into a document?

                          Vim 5.x: CTRL-V <decimal> byte
                          Vim 5.x: CTRL-V x <xx> byte
                          Vim 6.0: CTRL-V u <xxxx> 2 bytes
                          Vim 6.0: CTRL-V U <xxxxxxxx> 4 bytes

                          --
                          hundred-and-one symptoms of being an internet addict:
                          129. You cancel your newspaper subscription.

                          /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                          ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                          \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                        • Jürgen Krämer
                          Hi, ... and xff enters 0xff, too. Regards, Jürgen -- Jürgen Krämer Softwareentwicklung/-support Habel GmbH
                          Message 12 of 13 , May 7, 2001
                          • 0 Attachment
                            Hi,

                            Benji Fisher wrote:
                            >
                            > Vim is not a binary editor. It is a line-based editor. It needs
                            > some EOL character, and it uses 0x0a. You can enter a 0x0a by typing
                            > <CR> in Input mode. (I think I have this straight.) Any other
                            > character can be entered with <C-V>ddd. For example, <C-V>255 enters
                            > 0xff.

                            and <C-V>xff enters 0xff, too.

                            Regards,
                            Jürgen

                            --
                            Jürgen Krämer Softwareentwicklung/-support
                            Habel GmbH mailto:software@...
                            Hinteres Öschle 2 Tel: (0 74 61) 93 53 15
                            78604 Rietheim-Weilheim Fax: (0 74 61) 93 53 99
                          • Bram Moolenaar
                            Steve Amerige wrote ... You can define a region that starts at a match and ends at another match. Everything contained in that region can be different from
                            Message 13 of 13 , May 8, 2001
                            • 0 Attachment
                              Steve Amerige wrote

                              > Does VIM have the capability in it's syntax highlighting rules to:
                              >
                              > 1. know if it has scanned past a non-viewable character so as to be
                              > able to set some state variable;
                              >
                              > 2. know if it has scanned a specific token (e.g., "show") from a list
                              > of tokens (or rules) so as to be able to set some state variable; and,
                              >
                              > 3. use syntax rules based on some state variable?
                              >
                              > If it does then I could write syntax rules that take care of this kind
                              > of need. If not, do you think this would be something to consider
                              > for some future version of VIM?

                              You can define a region that starts at a match and ends at another match.
                              Everything contained in that region can be different from what is matched at
                              the top level. That should be sufficient to handle a sequence of characters
                              in the text that need different treatment.

                              --
                              Never eat yellow snow.

                              /// Bram Moolenaar -- Bram@... -- http://www.moolenaar.net \\\
                              ((( Creator of Vim -- http://vim.sf.net -- ftp://ftp.vim.org/pub/vim )))
                              \\\ Help me helping AIDS orphans in Uganda - http://iccf-holland.org ///
                            Your message has been successfully submitted and would be delivered to recipients shortly.