Loading ...
Sorry, an error occurred while loading the content.

Data structures allocated in shared memory

Expand Messages
  • Francois Berenger
    Hello, Is it possible to create some data structures so that they are allocated in a shared memory? Let s say an array of OCaml values for example. So that
    Message 1 of 4 , Nov 28, 2011
    • 0 Attachment
      Hello,

      Is it possible to create some data structures
      so that they are allocated in a shared memory?

      Let's say an array of OCaml values for example.
      So that this array can be written to by
      different processes without needing data copy
      back to the parent process once the children are finished
      with their job.

      Ideally, I am thinking the program would allocate
      a shared memory at the beginning, explicitly
      ask that some object is stored in it then
      release the shared memory once the object is no more used.

      Thanks a lot,
      F.
    • Gabriel Scherer
      Have a look at - the ancient module by Richard W. Jones: http://git.annexia.org/?p=ocaml-ancient.git;a=summary README:
      Message 2 of 4 , Nov 28, 2011
      • 0 Attachment
        Have a look at
        - the "ancient" module by Richard W. Jones:
        http://git.annexia.org/?p=ocaml-ancient.git;a=summary
        README: http://git.annexia.org/?p=ocaml-ancient.git;a=blob;f=README.txt
        - the new shared memory functions of Ocamlnet by Gerd Stolpmann
        http://projects.camlcity.org/projects/dl/ocamlnet-3.4.1/doc/html-main/Intro.html#netmulticore
        or the older 'netshm' module
        http://projects.camlcity.org/projects/dl/ocamlnet-3.4.1/doc/html-main/index.html#netshm

        On Tue, Nov 29, 2011 at 1:58 AM, Francois Berenger <berenger@...> wrote:
        > Hello,
        >
        > Is it possible to create some data structures
        > so that they are allocated in a shared memory?
        >
        > Let's say an array of OCaml values for example.
        > So that this array can be written to by
        > different processes without needing data copy
        > back to the parent process once the children are finished
        > with their job.
        >
        > Ideally, I am thinking the program would allocate
        > a shared memory at the beginning, explicitly
        > ask that some object is stored in it then
        > release the shared memory once the object is no more used.
        >
        > Thanks a lot,
        > F.
        >
        >
        > ------------------------------------
        >
        > Archives up to December 31, 2010 are also downloadable at http://www.connettivo.net/cntprojects/ocaml_beginners
        > The archives of the very official ocaml list (the seniors' one) can be found at http://caml.inria.fr
        > Attachments are banned and you're asked to be polite, avoid flames etc.Yahoo! Groups Links
        >
        >
        >
        >
      • Francois Berenger
        ... only read-only data structures ... too big for me ... maybe OK. Alternatively, I found this sentence on the web, which may do the trick without adding an
        Message 3 of 4 , Nov 28, 2011
        • 0 Attachment
          On 11/29/2011 02:54 PM, Gabriel Scherer wrote:
          > Have a look at
          > - the "ancient" module by Richard W. Jones:
          > http://git.annexia.org/?p=ocaml-ancient.git;a=summary
          > README: http://git.annexia.org/?p=ocaml-ancient.git;a=blob;f=README.txt

          only read-only data structures

          > - the new shared memory functions of Ocamlnet by Gerd Stolpmann
          > http://projects.camlcity.org/projects/dl/ocamlnet-3.4.1/doc/html-main/Intro.html#netmulticore
          > or the older 'netshm' module

          too big for me

          > http://projects.camlcity.org/projects/dl/ocamlnet-3.4.1/doc/html-main/index.html#netshm

          maybe OK.

          Alternatively, I found this sentence on the web, which may do the trick
          without adding an additional dependency:
          "- For sharing complex data, you can marshall into a shared Bigarray."

          Regards,
          F.

          > On Tue, Nov 29, 2011 at 1:58 AM, Francois Berenger <berenger@...
          > <mailto:berenger%40riken.jp>> wrote:
          > > Hello,
          > >
          > > Is it possible to create some data structures
          > > so that they are allocated in a shared memory?
          > >
          > > Let's say an array of OCaml values for example.
          > > So that this array can be written to by
          > > different processes without needing data copy
          > > back to the parent process once the children are finished
          > > with their job.
          > >
          > > Ideally, I am thinking the program would allocate
          > > a shared memory at the beginning, explicitly
          > > ask that some object is stored in it then
          > > release the shared memory once the object is no more used.
          > >
          > > Thanks a lot,
          > > F.
          > >
          > >
          > > ------------------------------------
          > >
          > > Archives up to December 31, 2010 are also downloadable at
          > http://www.connettivo.net/cntprojects/ocaml_beginners
          > > The archives of the very official ocaml list (the seniors' one) can
          > be found at http://caml.inria.fr
          > > Attachments are banned and you're asked to be polite, avoid flames
          > etc.Yahoo! Groups Links
          > >
          > >
          > >
          > >
          >
          >
        • Gabriel Scherer
          I believe the write-only restriction is meant to avoid data in the ancient heap to point to data in the OCaml heap (which could create a dangling ref if the
          Message 4 of 4 , Nov 29, 2011
          • 0 Attachment
            I believe the write-only restriction is meant to avoid data in the
            ancient heap to point to data in the OCaml heap (which could create a
            dangling ref if the data isn't also referred to in the OCaml heap).
            That means you can't manipulate data structure containing heap
            references; but I don't think mutating an int or float array would be
            a problem. Bigarray would also be a good idea if it suits your
            application better (in particular if you need to interact with C or
            Fortran code).

            Ocamlnet libraries probably have the same restrictions on shared data
            structures (nothing containing OCaml heap references).

            That said, if Ancient's documentation is formal than mutation is
            forbidden, it's probably not a good idea to go for it no matter what.
            I suspect however that by using mmapped bigarrays you will mostly
            re-implement a kind of Ancient module.

            On Tue, Nov 29, 2011 at 8:00 AM, Francois Berenger <berenger@...> wrote:
            > On 11/29/2011 02:54 PM, Gabriel Scherer wrote:
            >> Have a look at
            >> - the "ancient" module by Richard W. Jones:
            >> http://git.annexia.org/?p=ocaml-ancient.git;a=summary
            >> README: http://git.annexia.org/?p=ocaml-ancient.git;a=blob;f=README.txt
            >
            > only read-only data structures
            >
            >> - the new shared memory functions of Ocamlnet by Gerd Stolpmann
            >> http://projects.camlcity.org/projects/dl/ocamlnet-3.4.1/doc/html-main/Intro.html#netmulticore
            >> or the older 'netshm' module
            >
            > too big for me
            >
            >> http://projects.camlcity.org/projects/dl/ocamlnet-3.4.1/doc/html-main/index.html#netshm
            >
            > maybe OK.
            >
            > Alternatively, I found this sentence on the web, which may do the trick
            > without adding an additional dependency:
            > "- For sharing complex data, you can marshall into a shared Bigarray."
            >
            > Regards,
            > F.
            >
            >> On Tue, Nov 29, 2011 at 1:58 AM, Francois Berenger <berenger@...
            >> <mailto:berenger%40riken.jp>> wrote:
            >>  > Hello,
            >>  >
            >>  > Is it possible to create some data structures
            >>  > so that they are allocated in a shared memory?
            >>  >
            >>  > Let's say an array of OCaml values for example.
            >>  > So that this array can be written to by
            >>  > different processes without needing data copy
            >>  > back to the parent process once the children are finished
            >>  > with their job.
            >>  >
            >>  > Ideally, I am thinking the program would allocate
            >>  > a shared memory at the beginning, explicitly
            >>  > ask that some object is stored in it then
            >>  > release the shared memory once the object is no more used.
            >>  >
            >>  > Thanks a lot,
            >>  > F.
            >>  >
            >>  >
            >>  > ------------------------------------
            >>  >
            >>  > Archives up to December 31, 2010 are also downloadable at
            >> http://www.connettivo.net/cntprojects/ocaml_beginners
            >>  > The archives of the very official ocaml list (the seniors' one) can
            >> be found at http://caml.inria.fr
            >>  > Attachments are banned and you're asked to be polite, avoid flames
            >> etc.Yahoo! Groups Links
            >>  >
            >>  >
            >>  >
            >>  >
            >>
            >>
            >
            >
            >
            > ------------------------------------
            >
            > Archives up to December 31, 2010 are also downloadable at http://www.connettivo.net/cntprojects/ocaml_beginners
            > The archives of the very official ocaml list (the seniors' one) can be found at http://caml.inria.fr
            > Attachments are banned and you're asked to be polite, avoid flames etc.Yahoo! Groups Links
            >
            >
            >
          Your message has been successfully submitted and would be delivered to recipients shortly.