Loading ...
Sorry, an error occurred while loading the content.

12883Re: "ocaml_beginners"::[] Monitor memory usgae from executing ocaml code

Expand Messages
  • Johan Mazel
    Aug 1 11:27 AM
    • 0 Attachment
      Thanks for the advice.
      An example from the ocaml-benchmark project (
      https://forge.ocamlcore.org/projects/ocaml-benchmark/) show that I was
      wrong: array are faster than array1 especially when unsafe function are
      used.

      I therefore try to use arrays.
      Here is the result of the benchmark in term of memory usage obtained from
      pmap:
      Array: 2148,352MB
      Array1: 613,832MB
      The theoretic size should be 570MB (cf end of my mail) which is coherent
      with the value from Array1.

      However, the amount of memory used by array is way too big.
      A first explanation might be the fact that Ocaml's float are double
      precision float which would mean that they are supposed to occupy twice the
      space of float32 of Bigarray, ie 64 bits. Considering the measured memory
      use, this can not be the only reason.
      Another explanation might be the internal behavior/implementation of the
      array module. But I could not find any information on this specific point.

      Regards.
      Johan Mazel


      I am using array of one dimension to store triangular matrix as I've been
      advised in this thread :
      http://tech.groups.yahoo.com/group/ocaml_beginners/message/12840.
      The theoretical number of element in such array for 17296 points is
      (17296×(17296-1)÷2) = 149567160.
      Which occupy 149567160×32÷(8×(1024^2)) = 570,553436279 MB



      2011/8/1 Francois Berenger <berenger@...>

      > **
      >
      >
      > On 07/30/2011 11:41 PM, Johan Mazel wrote:
      > > My bad, this is THE "problem".
      > > I did not know that BigArray were allocated outside of the heap.
      > >
      > > I am going to keep using bigArray and call pmap from Ocaml in order to
      > get
      > > the memory usage estimation.
      > > I am afraid that a switch to Ocaml's array will slow down my code
      >
      > Benchmark, then you will really know.
      >
      >
      > > that
      > > heavily depends on accessing these arrays.
      > >
      > > Thanks a lot for your help.
      > > Regards.
      > > Johan mazel
      > >
      > > 2011/7/30 Gabriel Scherer<gabriel.scherer@...>
      > >
      > >> **
      > >>
      > >>
      > >> When you say Array2, Array1, are you talking about Bigarray.Array1 and
      > >> BigArray.Array2 modules ? If yes, they do live outside the OCaml heap.
      > If
      > >> you mean plain arrays ('a array, float array array; but I don't know any
      > >> module named Array1 or Array2 for these arrays) then there is something
      > >> else
      > >> going on.
      > >>
      > >>
      > >> On Sat, Jul 30, 2011 at 3:57 PM, Johan Mazel<johan.mazel@...>
      > >> wrote:
      > >>
      > >>> I do not think it is the case.
      > >>> The main part of this memory use is caused by either Array2 or Array1
      > and
      > >>> as
      > >>> far as I know these types are not used by Ocaml through C bindings.
      > >>> Regards.
      > >>> Johan Mazel
      > >>>
      > >>> 2011/7/30 Gabriel Scherer<gabriel.scherer@...>
      > >>>
      > >>>> **
      > >>>>
      > >>>>
      > >>>> The Gc will only count the memory allocated on the OCaml heap. If you
      > >> use
      > >>>> some OCaml/C binding, or Bigarray, you may well have large amounts of
      > >>>> memory
      > >>>> allocated outside the OCaml heap.
      > >>>>
      > >>>>
      > >>>> On Sat, Jul 30, 2011 at 2:16 PM, Johan Mazel<johan.mazel@...>
      > >>>> wrote:
      > >>>>
      > >>>>> Hi
      > >>>>> I am trying to asses the memory usage of an Ocaml program in order to
      > >>>>> benchmark its execution.
      > >>>>>
      > >>>>> I tried to use the heap_words field from Gc stat as stated here
      > >>>>> http://tech.groups.yahoo.com/group/ocaml_beginners/message/3378 but
      > >>>> there
      > >>>>> is
      > >>>>> a discrepancy between this value and the one that pmap gives me on my
      > >>>>> process.
      > >>>>> In fact, heap_words give me a value 507904 which is B/MB
      > >>>>> 507904×64÷8=4063232B or 3.875MB (64 bits architecture) when pmap
      > >> gives
      > >>> me
      > >>>> a
      > >>>>> value of 211528kB×1024=216604672B or 206MB.
      > >>>>> Pmap's value is much more consistent with top's value which is 4.7%
      > >> of
      > >>>>> total
      > >>>>> RAM => 4058692×1024×0.047=195336728,576B or 186MB (the error can
      > >>>> partially
      > >>>>> be explained by the lack of precision of the 4.7%).
      > >>>>>
      > >>>>> One of the first explanation that I found was that maybe this error
      > >> was
      > >>>>> linked to the fact that heap_words only concerns words in the major
      > >>> heap
      > >>>>> and
      > >>>>> not in the minor heap. However, the error is so big that in order for
      > >>>> this
      > >>>>> to be the real reason of my problem, the minor heap should be bigger
      > >>> that
      > >>>>> the major one which impossible if I understood correctly how the GC
      > >>>> works.
      > >>>>> I think that I am missing something but I don't see what it is.
      > >>>>>
      > >>>>> Thanks in advance for your time.
      > >>>>> Regards.
      > >>>>> Johan Mazel
      > >>>>>
      > >>>>>
      > >>>>> [Non-text portions of this message have been removed]
      > >>>>>
      > >>>>>
      > >>>>>
      > >>>>> ------------------------------------
      > >>>>
      > >>>>>
      > >>>>> Archives up to December 31, 2010 are also downloadable at
      > >>>>> http://www.connettivo.net/cntprojects/ocaml_beginners
      > >>>>> The archives of the very official ocaml list (the seniors' one) can
      > >> be
      > >>>>> found at http://caml.inria.fr
      > >>>>> Attachments are banned and you're asked to be polite, avoid flames
      > >>>>> etc.Yahoo! Groups Links
      > >>>>
      > >>>>>
      > >>>>>
      > >>>>>
      > >>>>>
      > >>>>
      > >>>> [Non-text portions of this message have been removed]
      > >>>>
      > >>>>
      > >>>>
      > >>>
      > >>>
      > >>> [Non-text portions of this message have been removed]
      > >>>
      > >>>
      > >>>
      > >>> ------------------------------------
      > >>>
      > >>> Archives up to December 31, 2010 are also downloadable at
      > >>> http://www.connettivo.net/cntprojects/ocaml_beginners
      > >>> The archives of the very official ocaml list (the seniors' one) can be
      > >>> found at http://caml.inria.fr
      > >>> Attachments are banned and you're asked to be polite, avoid flames
      > >>> etc.Yahoo! Groups Links
      > >>>
      > >>>
      > >>>
      > >>>
      > >>
      > >> [Non-text portions of this message have been removed]
      > >>
      > >>
      > >>
      > >
      > >
      > > [Non-text portions of this message have been removed]
      > >
      > >
      > >
      > > ------------------------------------
      > >
      > > Archives up to December 31, 2010 are also downloadable at
      > http://www.connettivo.net/cntprojects/ocaml_beginners
      > > The archives of the very official ocaml list (the seniors' one) can be
      > found at http://caml.inria.fr
      > > Attachments are banned and you're asked to be polite, avoid flames
      > etc.Yahoo! Groups Links
      > >
      > >
      > >
      >
      >
      >


      [Non-text portions of this message have been removed]
    • Show all 11 messages in this topic