Loading ...
Sorry, an error occurred while loading the content.

Re: [Io] Object system streams

Expand Messages
  • Kentaro A. Kurahone
    On Tue, 30 Sep 2003 23:20:03 +0000 ... Famous last words. I have a patch that will work on Sane Systems(TM). Unfortunately in this context, the Sane
    Message 1 of 24 , Oct 1, 2003
    View Source
    • 0 Attachment
      On Tue, 30 Sep 2003 23:20:03 +0000
      > Hmmm, should be easy enough to add. I'll poke at it over the next few
      > days. IoL4's process creation code is giving me a headache, and a bit
      > of rest will help (New VM is coded and done, yay). ;)
      Famous last words. I have a patch that will work on Sane Systems(TM).
      Unfortunately in this context, the Sane System(TM) does not include
      Linux as their popen implementation does not handle bidirectional
      communication (FreeBSD/OSX work fine).

      So the net effect is that under Linux, you can have either access to
      stdin or access to stdout, but not both.

      Yes, there is a workaround, but it involves large reworks to IoFile.
      Changing it from using file pointers to their file descriptor
      equivalents, storing a bit more state, and coding up a implementation of
      popen/pclose. Should only take a hour or so (if that) to code up, but
      there's a question of if it's worth it.

      Thoughts?

      --
      Kentaro A. Kurahone
      SIGUSR1 Research and Development
      "If you would excuse me, I'm in the middle of 15 different things all
      annoying."
    • Tobias Peters
      ... Can you work around that on a higher level, in pure Io, by creating two pipes, and have a delegator object dispatch methods to the proper pipe? Tobias
      Message 2 of 24 , Oct 1, 2003
      View Source
      • 0 Attachment
        Kentaro A. Kurahone wrote:
        > So the net effect is that under Linux, you can have either access to
        > stdin or access to stdout, but not both.
        >
        > Yes, there is a workaround, but it involves large reworks to IoFile.
        > Changing it from using file pointers to their file descriptor
        > equivalents, storing a bit more state, and coding up a implementation of
        > popen/pclose. Should only take a hour or so (if that) to code up, but
        > there's a question of if it's worth it.
        >
        > Thoughts?

        Can you work around that on a higher level, in pure Io, by creating two
        pipes, and have a delegator object dispatch methods to the proper pipe?

        Tobias
      • Kentaro A. Kurahone
        On Wed, 01 Oct 2003 13:20:41 +0200 ... Hmmm, don t think so. The problem is that with Linux popen, it appears that you only get one single unidirectional
        Message 3 of 24 , Oct 1, 2003
        View Source
        • 0 Attachment
          On Wed, 01 Oct 2003 13:20:41 +0200
          Tobias Peters <tpeters@...> wrote:
          > Kentaro A. Kurahone wrote:
          > > So the net effect is that under Linux, you can have either access to
          > > stdin or access to stdout, but not both.
          > >
          > > Yes, there is a workaround, but it involves large reworks to IoFile.
          > > Changing it from using file pointers to their file descriptor
          > > equivalents, storing a bit more state, and coding up a
          > > implementation of popen/pclose. Should only take a hour or so (if
          > > that) to code up, but there's a question of if it's worth it.
          > >
          > > Thoughts?
          >
          > Can you work around that on a higher level, in pure Io, by creating
          > two pipes, and have a delegator object dispatch methods to the proper
          > pipe?
          Hmmm, don't think so. The problem is that with Linux popen, it appears
          that you only get one single unidirectional endpoint. So it'll still
          mean reimplementing a saner popen in C land, and if you're going to go
          to that effort, might as well make IoFile work off fds instead of file
          pointers. (It naturally is perfectly permissible to say "Tough" and
          make the popen call return a read only IoFile object. It'll be
          functionally equivalent to the `cmd` operator in perl/shell script.)

          But I just finished the code changes to IoFile and reimplemented the
          popen method (see attached patch). I did test basic file stuff for
          regressions, but there's probably something I missed.

          As a sideeffect readLine is significantly slower now, since it seems
          neccecary to read one character at a time. (can't seek on a pipe.)
          Could read in large chunks and buffer the output, but that'll increase
          code complexity considerably. I suppose it's possible to have 2
          versions of ByteArray_readLineFromCStream_, one for pipes, one for non.

          Also since I'm using fork, execv, and assuming that there's a file named
          /bin/sh, this probably breakes flavors of win32 though under cygwin I
          assume that it'll happily execute bash.

          --
          Kentaro A. Kurahone
          SIGUSR1 Research and Development
          "If you would excuse me, I'm in the middle of 15 different things all
          annoying."

          [0]: I guess it's really getting to be time to do the config/build
          system rework. Autoconf/automake, or a beefed up config
          makefile/shellscript of lesser doom.
        • Chris Double
          ... Will this be in IoServer and not IoVM? Adding this to IoVM will also break portability to Symbian and other embedded OS s. Chris.
          Message 4 of 24 , Oct 1, 2003
          View Source
          • 0 Attachment
            On Wed, 01 Oct 2003 23:43, Kentaro A. Kurahone wrote:
            > Also since I'm using fork, execv, and assuming that there's a file named
            > /bin/sh, this probably breakes flavors of win32 though under cygwin I
            > assume that it'll happily execute bash.

            Will this be in IoServer and not IoVM? Adding this to IoVM will also break
            portability to Symbian and other embedded OS's.

            Chris.
          • Kentaro A. Kurahone
            On Thu, 2 Oct 2003 01:11:31 +1200 ... Possible, yes. Is Symbian libc lacking POSIX file desscriptor related functions or just fork/execv? Yours, -- Kentaro A.
            Message 5 of 24 , Oct 1, 2003
            View Source
            • 0 Attachment
              On Thu, 2 Oct 2003 01:11:31 +1200
              Chris Double <chris.double@...> wrote:
              > On Wed, 01 Oct 2003 23:43, Kentaro A. Kurahone wrote:
              > > Also since I'm using fork, execv, and assuming that there's a file
              > > named/bin/sh, this probably breakes flavors of win32 though under
              > > cygwin I assume that it'll happily execute bash.
              >
              > Will this be in IoServer and not IoVM? Adding this to IoVM will also
              > break portability to Symbian and other embedded OS's.
              Possible, yes. Is Symbian libc lacking POSIX file desscriptor related
              functions or just fork/execv?

              Yours,

              --
              Kentaro A. Kurahone
              SIGUSR1 Research and Development
              "If you would excuse me, I'm in the middle of 15 different things all
              annoying."
            • Steve Dekorte
              ... No worries - it won t go in IoVM unless it s portable. And wherever it does go, it will need to be easily ifdefed out if not supported. -- Steve
              Message 6 of 24 , Oct 1, 2003
              View Source
              • 0 Attachment
                On Wednesday, October 1, 2003, at 6:11 AM, Chris Double wrote:
                > Will this be in IoServer and not IoVM? Adding this to IoVM will also
                > break
                > portability to Symbian and other embedded OS's.

                No worries - it won't go in IoVM unless it's portable. And wherever it
                does go, it will need to be easily ifdefed out if not supported.

                -- Steve
              • Chris Double
                On Wed, 1 Oct 2003 18:30:33 +0000, Kentaro A. Kurahone ... It definitely doesn t have fork/execv. I m pretty sure it doesn t have the POSIX file descriptor
                Message 7 of 24 , Oct 1, 2003
                View Source
                • 0 Attachment
                  On Wed, 1 Oct 2003 18:30:33 +0000, "Kentaro A. Kurahone"
                  <kurahone@...> said:
                  > Possible, yes. Is Symbian libc lacking POSIX file desscriptor related
                  > functions or just fork/execv?

                  It definitely doesn't have fork/execv. I'm pretty sure it doesn't have
                  the POSIX file descriptor stuff, only stdio.

                  Chris.
                  --
                  Chris Double
                  chris.double@...
                • Kentaro A. Kurahone
                  On Thu, 02 Oct 2003 09:53:45 +1200 ... Hmmm. Their documentation references a fairly complete unistd.h as long as rev 5.0 of the library is being used... I
                  Message 8 of 24 , Oct 1, 2003
                  View Source
                  • 0 Attachment
                    On Thu, 02 Oct 2003 09:53:45 +1200
                    "Chris Double" <chris.double@...> wrote:
                    > On Wed, 1 Oct 2003 18:30:33 +0000, "Kentaro A. Kurahone"
                    > <kurahone@...> said:
                    > > Possible, yes. Is Symbian libc lacking POSIX file desscriptor
                    > > related functions or just fork/execv?
                    >
                    > It definitely doesn't have fork/execv. I'm pretty sure it doesn't have
                    > the POSIX file descriptor stuff, only stdio.

                    Hmmm. Their documentation references a fairly complete unistd.h as long
                    as rev 5.0 of the library is being used... I might be looking at the
                    wrong place though. Never did any devel for their system.

                    http://www.symbian.com/developer/techlib/v70docs/SDL_v7.0/doc_source/reference/cpp/libc/sys/unistd_h/


                    --
                    Kentaro A. Kurahone
                    SIGUSR1 Research and Development
                    "If you would excuse me, I'm in the middle of 15 different things all
                    annoying."
                  • Chris Double
                    On Wed, 1 Oct 2003 22:00:34 +0000, Kentaro A. Kurahone ... Yes, it looks like they do support the POSIX file descriptor stuff but not fork.exec:
                    Message 9 of 24 , Oct 1, 2003
                    View Source
                    • 0 Attachment
                      On Wed, 1 Oct 2003 22:00:34 +0000, "Kentaro A. Kurahone"
                      <kurahone@...> said:
                      > Hmmm. Their documentation references a fairly complete unistd.h as long
                      > as rev 5.0 of the library is being used... I might be looking at the
                      > wrong place though. Never did any devel for their system.

                      Yes, it looks like they do support the POSIX file descriptor stuff but
                      not fork.exec:

                      http://www.symbian.com/developer/techlib/v70docs/SDL_v7.0/doc_source/DevGuides/cpp/Base/CStandardLibrary/DesignSTDLIB.html

                      (Under Exclusions at the end it lists fork, exec and select as not being
                      implemented)

                      Chris.
                      --
                      Chris Double
                      chris.double@...
                    • Daniel Wunsch
                      ... it s a pity, really. i wonder how the java-guys implemented their streams on symbian.. regards, daniel
                      Message 10 of 24 , Oct 1, 2003
                      View Source
                      • 0 Attachment
                        On Thursday 02 October 2003 00:08, Chris Double wrote:

                        > (Under Exclusions at the end it lists fork, exec and select as not being
                        > implemented)

                        it's a pity, really. i wonder how the java-guys
                        implemented their streams on symbian..

                        regards,
                        daniel
                      • Daniel Wunsch
                        ... i think the speedup resulting from reading larger chunks would be worth the effort - i m not even sure if a file should know about the concept of a line,
                        Message 11 of 24 , Oct 1, 2003
                        View Source
                        • 0 Attachment
                          On Wednesday 01 October 2003 13:43, Kentaro A. Kurahone wrote:

                          > As a sideeffect readLine is significantly slower now, since it seems
                          > neccecary to read one character at a time. (can't seek on a pipe.)
                          > Could read in large chunks and buffer the output, but that'll increase
                          > code complexity considerably.

                          i think the speedup resulting from reading larger chunks would
                          be worth the effort - i'm not even sure if a file should know
                          about the concept of a line, because that clearly is a problem
                          in the string domain..

                          regards,
                          daniel
                        • Chris Double
                          On Thu, 2 Oct 2003 01:33:27 +0200, Daniel Wunsch ... They would have implemented that sort of thing using the Symbian API rather than the
                          Message 12 of 24 , Oct 1, 2003
                          View Source
                          • 0 Attachment
                            On Thu, 2 Oct 2003 01:33:27 +0200, "Daniel Wunsch" <the.gray@...>
                            said:
                            > it's a pity, really. i wonder how the java-guys
                            > implemented their streams on symbian..

                            They would have implemented that sort of thing using the Symbian API
                            rather than the standard C/Posix wrapper library. That library is really
                            just used for porting existing programs while they can be migrated to the
                            Symbian API which includes lots of process/thread type functionality.

                            The Symbian API is very full featured but it is all C++ and insists on
                            being the 'controller'. All 'applications' are really DLL's that are
                            loaded by the main executable which drives everything.

                            Chris.
                            --
                            Chris Double
                            chris.double@...
                          • Daniel Wunsch
                            ... hm.. an (ugly) way to solve this issue could be running a native symbian program using the symbian API and make it communicate with a piece of code
                            Message 13 of 24 , Oct 1, 2003
                            View Source
                            • 0 Attachment
                              On Thursday 02 October 2003 01:38, Chris Double wrote:

                              > The Symbian API is very full featured but it is all C++ and insists on
                              > being the 'controller'. All 'applications' are really DLL's that are
                              > loaded by the main executable which drives everything.

                              hm.. an (ugly) way to solve this issue could be running a
                              'native' symbian program using the symbian API and make
                              it communicate with a piece of code emulating the posix
                              functionality..

                              problem is, this would lead to even more #ifdefs in the code :/

                              daniel
                            • Steve Dekorte
                              ... Personally, I don t like pipes and would just assume not directly support them unless necessary. Did the original application that this was requested for
                              Message 14 of 24 , Oct 1, 2003
                              View Source
                              • 0 Attachment
                                On Wednesday, October 1, 2003, at 4:49 PM, Daniel Wunsch wrote:
                                > hm.. an (ugly) way to solve this issue could be running a
                                > 'native' symbian program using the symbian API and make
                                > it communicate with a piece of code emulating the posix
                                > functionality..
                                >
                                > problem is, this would lead to even more #ifdefs in the code :/

                                Personally, I don't like pipes and would just assume not directly
                                support them unless necessary. Did the original application that this
                                was requested for actually need them? That is, was piping to a file and
                                then reading the file an unworkable solution?

                                -- Steve
                              • Daniel Wunsch
                                ... being a unix admin, i like pipes very much, because they so extremely use- and powerful. ... to say it needs them would be too strong.. ... using a
                                Message 15 of 24 , Oct 1, 2003
                                View Source
                                • 0 Attachment
                                  On Thursday 02 October 2003 02:14, Steve Dekorte wrote:

                                  > Personally, I don't like pipes and would just assume not directly
                                  > support them unless necessary.

                                  being a unix admin, i like pipes very much,
                                  because they so extremely use- and powerful.

                                  > Did the original application that this
                                  > was requested for actually need them?

                                  to say it needs them would be too strong..

                                  > That is, was piping to a file and
                                  > then reading the file an unworkable solution?

                                  using a temporary file does work, but personally
                                  i never liked tempfiles - they tend to fill up
                                  filesystems if a program crashes, gets aborted
                                  or the programmer forgets to delete them.

                                  and: you have to make sure you don't make
                                  your code vulnerable to symlink attacks.

                                  talking about tempfiles: i found this line
                                  in my compiler output, but i don't know
                                  whethe my GCC is right here, since
                                  i didn't look in the sourcecode.

                                  _libs/libIoVM.a(IoFile.o)(.text+0x765): In function `IoFile_useTemporaryPath':
                                  /home/daniel/io/IoDesktop-2003-09-23/IoVM/IoFile.c:224: warning: the use of
                                  `tmpnam' is dangerous, better use `mkstemp'

                                  regards,
                                  daniel
                                • Kentaro A. Kurahone
                                  On Thu, 2 Oct 2003 02:31:13 +0200 ... Ditto. Personally I think that it ll be nice to have a Io equivalent of the `cmd` operator present in perl/sh/etc, if
                                  Message 16 of 24 , Oct 1, 2003
                                  View Source
                                  • 0 Attachment
                                    On Thu, 2 Oct 2003 02:31:13 +0200
                                    Daniel Wunsch <the.gray@...> wrote:
                                    > On Thursday 02 October 2003 02:14, Steve Dekorte wrote:
                                    >
                                    > > Personally, I don't like pipes and would just assume not directly
                                    > > support them unless necessary.
                                    >
                                    > being a unix admin, i like pipes very much,
                                    > because they so extremely use- and powerful.
                                    >
                                    > > Did the original application that this
                                    > > was requested for actually need them?
                                    >
                                    > to say it needs them would be too strong..
                                    >
                                    > > That is, was piping to a file and
                                    > > then reading the file an unworkable solution?
                                    >
                                    > using a temporary file does work, but personally
                                    > i never liked tempfiles - they tend to fill up
                                    > filesystems if a program crashes, gets aborted
                                    > or the programmer forgets to delete them.
                                    >
                                    > and: you have to make sure you don't make
                                    > your code vulnerable to symlink attacks.
                                    Ditto. Personally I think that it'll be nice to have a Io equivalent of
                                    the "`cmd`" operator present in perl/sh/etc, if that's all that needed
                                    (no need to write to stdin), popen without changes to file descriptor
                                    should work on most unix variants.

                                    Yours,

                                    --
                                    Kentaro A. Kurahone
                                    SIGUSR1 Research and Development
                                    "If you would excuse me, I'm in the middle of 15 different things all
                                    annoying."
                                  • Steve Dekorte
                                    ... Would you say they are the ideal mechanism for coupling processes? What I m getting at is that I d prefer to see processes replaced with objects, and pipes
                                    Message 17 of 24 , Oct 2, 2003
                                    View Source
                                    • 0 Attachment
                                      On Wednesday, October 1, 2003, at 5:31 PM, Daniel Wunsch wrote:
                                      > On Thursday 02 October 2003 02:14, Steve Dekorte wrote:
                                      >> Personally, I don't like pipes and would just assume not directly
                                      >> support them unless necessary.
                                      >
                                      > being a unix admin, i like pipes very much,
                                      > because they so extremely use- and powerful.

                                      Would you say they are the ideal mechanism for coupling processes?

                                      What I'm getting at is that I'd prefer to see processes replaced with
                                      objects, and pipes replaced with messages. This way all levels of
                                      communication and abstraction are unified into a single, flexible
                                      model.

                                      > _libs/libIoVM.a(IoFile.o)(.text+0x765): In function
                                      > `IoFile_useTemporaryPath':
                                      > /home/daniel/io/IoDesktop-2003-09-23/IoVM/IoFile.c:224: warning: the
                                      > use of
                                      > `tmpnam' is dangerous, better use `mkstemp'

                                      I think this is an ifdef issue.

                                      -- Steve
                                    • Daniel Wunsch
                                      ... processes in general? no. but in the unix-domain there already are a great number of useful programs do communicate via pipes, and i learned one thing from
                                      Message 18 of 24 , Oct 2, 2003
                                      View Source
                                      • 0 Attachment
                                        On Thursday 02 October 2003 11:43, Steve Dekorte wrote:
                                        > On Wednesday, October 1, 2003, at 5:31 PM, Daniel Wunsch wrote:

                                        > > being a unix admin, i like pipes very much,
                                        > > because they so extremely use- and powerful.
                                        >
                                        > Would you say they are the ideal mechanism for coupling processes?

                                        processes in general? no. but in the unix-domain there already are
                                        a great number of useful programs do communicate via pipes,
                                        and i learned one thing from heavy shell coding the last year:
                                        using pipes is often awkward and exhausting - but as a reward
                                        you can try out (and use) very powerful software without having to
                                        implement a binding to some obscure library first and so you can
                                        try different ways to solve a complex problem much faster.

                                        > What I'm getting at is that I'd prefer to see processes replaced with
                                        > objects, and pipes replaced with messages. This way all levels of
                                        > communication and abstraction are unified into a single, flexible
                                        > model.

                                        unifying them sounds nice, but i fear the performance penalty would
                                        be too high. sure, you could implement a real byte-stream (which as
                                        i explained before, many existing programs understand) in term
                                        as one-message-per-byte, but no way that ever will be fast enough
                                        for real-world problems.

                                        i prefer to see streams as a low-level thing, and a message
                                        messages as objects that _can_ be marshalled into streams.

                                        daniel
                                      • Steve Dekorte
                                        ... I agree that there already exist tools that use this method of communication that people find useful. However, if we can see a significantly better way to
                                        Message 19 of 24 , Oct 2, 2003
                                        View Source
                                        • 0 Attachment
                                          On Thursday, October 2, 2003, at 12:39 PM, Daniel Wunsch wrote:
                                          > On Thursday 02 October 2003 11:43, Steve Dekorte wrote:
                                          >> Would you say they are the ideal mechanism for coupling processes?
                                          >
                                          > processes in general? no. but in the unix-domain there already are
                                          > a great number of useful programs do communicate via pipes,
                                          > and i learned one thing from heavy shell coding the last year:
                                          > using pipes is often awkward and exhausting - but as a reward
                                          > you can try out (and use) very powerful software without having to
                                          > implement a binding to some obscure library first and so you can
                                          > try different ways to solve a complex problem much faster.

                                          I agree that there already exist tools that use this method of
                                          communication that people find useful. However, if we can see a
                                          significantly better way to do this, then it follows that pipes a poor
                                          method of communication in comparison.

                                          >> What I'm getting at is that I'd prefer to see processes replaced with
                                          >> objects, and pipes replaced with messages. This way all levels of
                                          >> communication and abstraction are unified into a single, flexible
                                          >> model.
                                          >
                                          > unifying them sounds nice, but i fear the performance penalty would
                                          > be too high. sure, you could implement a real byte-stream (which as
                                          > i explained before, many existing programs understand) in term
                                          > as one-message-per-byte, but no way that ever will be fast enough
                                          > for real-world problems.

                                          Performance is one of the advantages. Many programs that pipe data
                                          between them end up heavily re-parsing the data since it arrives in an
                                          unstructured form. I've written many unix scripts that had to pipe text
                                          streams between awk, sed, grep, etc and at least in my experience, much
                                          work was repeated and the resulting algorithms where cumbersome and
                                          fragile.

                                          Imagine the power that would come of both keeping all our data in
                                          simple unified object structures (instead of every unix program having
                                          it's own configuration language, log format, etc) and being able to
                                          pass objects around instead of text streams. I suspect that you could
                                          do everything that unix can do in a way that was far smaller, faster
                                          and simpler.

                                          -- Steve
                                        • Daniel Wunsch
                                          ... i agree on that for communication between newly-written applications. but i d very much like to use io as a scripting language in the unix-domain - and
                                          Message 20 of 24 , Oct 6, 2003
                                          View Source
                                          • 0 Attachment
                                            On Friday 03 October 2003 00:21, Steve Dekorte wrote:

                                            > I agree that there already exist tools that use this method of
                                            > communication that people find useful. However, if we can see a
                                            > significantly better way to do this, then it follows that pipes a poor
                                            > method of communication in comparison.

                                            i agree on that for communication between newly-written
                                            applications. but i'd very much like to use io as a scripting
                                            language in the unix-domain - and there, pipes are the way
                                            to do it.

                                            > Performance is one of the advantages. Many programs that pipe data
                                            > between them end up heavily re-parsing the data since it arrives in an
                                            > unstructured form.

                                            hm.. your have a point ;)

                                            > I've written many unix scripts that had to pipe text
                                            > streams between awk, sed, grep, etc and at least in my experience, much
                                            > work was repeated and the resulting algorithms where cumbersome and
                                            > fragile.

                                            true. but in some cases (shell scripting..) i prefer parsing in plain
                                            io over writing down a specification how the stream looks like first,
                                            mainly because that means to learn yet-another-language,
                                            even if it's expressed in io objects.

                                            > Imagine the power that would come of both keeping all our data in
                                            > simple unified object structures (instead of every unix program having
                                            > it's own configuration language, log format, etc) and being able to
                                            > pass objects around instead of text streams. I suspect that you could
                                            > do everything that unix can do in a way that was far smaller, faster
                                            > and simpler.

                                            whow - that would be very, very nice. problem is: nobody will
                                            rewrite apache, mySQL, tomcat, jEdit, kMail, mozilla, vim and
                                            all the other nice tools for me, so id' have to write a grammar
                                            describing their existing formats to use them..

                                            further, you have to differentiate between performance-optimized
                                            data formats for communication of high data volumes and simple
                                            low-volume structures like config-files, logs and so on.

                                            for the latter, XML seems to work quite ok, for the rest providing
                                            a grammar (have you seen, what perl 6 makes of regexps?
                                            faszinating!) and having the parser generated automtically would
                                            be a very useful feature, but since more and more communication
                                            is done in XML, maybe plain byte-arrays are good enough..

                                            daniel
                                          Your message has been successfully submitted and would be delivered to recipients shortly.