Loading ...
Sorry, an error occurred while loading the content.
 

parallel List.map

Expand Messages
  • Francois Berenger
    Hello, I d like to process a list in parallel on a multi-core machine. I don t care that the resulting list elements are disordered after this operation
    Message 1 of 7 , Feb 25, 2011
      Hello,

      I'd like to process a list in parallel on a multi-core machine.

      I don't care that the resulting list elements are disordered
      after this operation compared to the input list order.

      Is there such thing in Ocaml?

      If there is some choice, I prefer a multi process solution over a
      thread-based one. Also, I'd like to control the number of processes
      involved.

      Thanks,
      F.
    • Sylvain Le Gall
      ... Though I don t yet try it, there is plasma: http://plasma.camlcity.org/plasma/index.html This is maybe bigger than what you expect (because it can handle
      Message 2 of 7 , Feb 25, 2011
        On 25-02-2011, Francois Berenger <berenger@...> wrote:
        > --Jf1dPIii0QOEffl50VqGbDs0KbC-ijwCtUt8snj
        > Content-Type: text/plain; charset=ISO-8859-1
        > Content-Transfer-Encoding: 7bit
        >
        > Hello,
        >
        > I'd like to process a list in parallel on a multi-core machine.
        >
        > I don't care that the resulting list elements are disordered
        > after this operation compared to the input list order.
        >
        > Is there such thing in Ocaml?
        >
        > If there is some choice, I prefer a multi process solution over a
        > thread-based one. Also, I'd like to control the number of processes
        > involved.
        >

        Though I don't yet try it, there is plasma:
        http://plasma.camlcity.org/plasma/index.html

        This is maybe bigger than what you expect (because it can handle
        multiple machine), but maybe it is worth a try.

        Cheers,
        Sylvain Le Gall
        --
        My company: http://www.ocamlcore.com
        Linkedin: http://fr.linkedin.com/in/sylvainlegall
        Start an OCaml project here: http://forge.ocamlcore.org
        OCaml blogs: http://planet.ocamlcore.org
      • Virgile Prevosto
        Hello, ... I haven t really used it myself, but you might be interested by the Functory library: http://www.lri.fr/~kalyan/functory/About.html -- E tutto per
        Message 3 of 7 , Feb 25, 2011
          Hello,

          2011/2/25 Francois Berenger <berenger@...>:
          > I'd like to process a list in parallel on a multi-core machine.
          >
          > I don't care that the resulting list elements are disordered
          > after this operation compared to the input list order.
          >
          > Is there such thing in Ocaml?
          >

          I haven't really used it myself, but you might be interested by the
          Functory library:
          http://www.lri.fr/~kalyan/functory/About.html

          --
          E tutto per oggi, a la prossima volta
          Virgile
        • Francois Berenger
          Hi, Thanks for all the answers. I don t need all the bells and whistles. Just parallelizing on the local cores would be enough. So, no parallelization over the
          Message 4 of 7 , Feb 25, 2011
            Hi,

            Thanks for all the answers.

            I don't need all the bells and whistles.
            Just parallelizing on the local cores would be enough.
            So, no parallelization over the network needed.

            I don't need (yet?) Erlang or MapReduce.

            On a second thought, the input might be a table
            as well as the output to allow in-place modifications.
            In which case the resulting table will be correctly ordered.

            Regards,
            F.

            Christophe TROESTLER wrote:
            > On Fri, 25 Feb 2011 17:53:32 +0900, Francois Berenger wrote:
            >> Hello,
            >>
            >> I'd like to process a list in parallel on a multi-core machine.
            >>
            >> I don't care that the resulting list elements are disordered
            >> after this operation compared to the input list order.
            >>
            >> Is there such thing in Ocaml?
            >>
            >> If there is some choice, I prefer a multi process solution over a
            >> thread-based one. Also, I'd like to control the number of processes
            >> involved.
            >
            > http://www.lri.fr/~filliatr/functory/About.html
          • Francois Berenger
            Hi, Functory looks one nice way to go. Anyone has a working example to use it in a project which uses ocamlbuild? Thanks, F.
            Message 5 of 7 , Feb 25, 2011
              Hi,

              Functory looks one nice way to go.

              Anyone has a working example to use it in a project
              which uses ocamlbuild?

              Thanks,
              F.

              Francois Berenger wrote:
              >
              >
              > Hi,
              >
              > Thanks for all the answers.
              >
              > I don't need all the bells and whistles.
              > Just parallelizing on the local cores would be enough.
              > So, no parallelization over the network needed.
              >
              > I don't need (yet?) Erlang or MapReduce.
              >
              > On a second thought, the input might be a table
              > as well as the output to allow in-place modifications.
              > In which case the resulting table will be correctly ordered.
              >
              > Regards,
              > F.
              >
              > Christophe TROESTLER wrote:
              > > On Fri, 25 Feb 2011 17:53:32 +0900, Francois Berenger wrote:
              > >> Hello,
              > >>
              > >> I'd like to process a list in parallel on a multi-core machine.
              > >>
              > >> I don't care that the resulting list elements are disordered
              > >> after this operation compared to the input list order.
              > >>
              > >> Is there such thing in Ocaml?
              > >>
              > >> If there is some choice, I prefer a multi process solution over a
              > >> thread-based one. Also, I'd like to control the number of processes
              > >> involved.
              > >
              > > http://www.lri.fr/~filliatr/functory/About.html
            • Michael Ekstrand
              ... Unfortunately, OCaml s current runtime has a lock preventing a single process from using more than one core in parallel (except for some code called
              Message 6 of 7 , Feb 25, 2011
                On 02/25/2011 03:39 AM, Francois Berenger wrote:
                >
                >
                > Hi,
                >
                > Thanks for all the answers.
                >
                > I don't need all the bells and whistles.
                > Just parallelizing on the local cores would be enough.
                > So, no parallelization over the network needed.
                >

                Unfortunately, OCaml's current runtime has a lock preventing a single
                process from using more than one core in parallel (except for some code
                called through C stubs). So you'll need some sort of multi-process
                parallelization solution. Plasma provides one that also extends across
                machines. You can also look at OCamlnet 3 - it has a parallelism
                framework that dispatches jobs across multiple processes. I believe it
                uses shared memory to do it, but I am not sure.

                - Michael


                [Non-text portions of this message have been removed]
              • Francois Berenger
                OK, with Functory I could parallelize the List.map I was interested into. It looks really a nice technology. My computer and all its cores are really busy now.
                Message 7 of 7 , Feb 28, 2011
                  OK, with Functory I could parallelize the List.map I was interested into.

                  It looks really a nice technology. My computer and all its cores are
                  really busy now. :)
                  While my code almost did not change.

                  I just followed this tutorial:
                  http://www.lri.fr/~kalyan/functory/Mini-Tutorial.html

                  And added these options to ocamlbuild (thanks to
                  Jean-Christophe Filliatre, LRI, France):
                  ---
                  ocamlbuild -cflags -I,+functory -lflags
                  unix.cmxa,-I,+functory,functory.cmxa main.native
                  ---

                  Regards,
                  F.

                  Michael Ekstrand wrote:
                  >
                  >
                  > On 02/25/2011 03:39 AM, Francois Berenger wrote:
                  > >
                  > >
                  > > Hi,
                  > >
                  > > Thanks for all the answers.
                  > >
                  > > I don't need all the bells and whistles.
                  > > Just parallelizing on the local cores would be enough.
                  > > So, no parallelization over the network needed.
                  > >
                  >
                  > Unfortunately, OCaml's current runtime has a lock preventing a single
                  > process from using more than one core in parallel (except for some code
                  > called through C stubs). So you'll need some sort of multi-process
                  > parallelization solution. Plasma provides one that also extends across
                  > machines. You can also look at OCamlnet 3 - it has a parallelism
                  > framework that dispatches jobs across multiple processes. I believe it
                  > uses shared memory to do it, but I am not sure.
                  >
                  > - Michael
                  >
                  > [Non-text portions of this message have been removed]
                  >
                  >
                Your message has been successfully submitted and would be delivered to recipients shortly.