Loading ...
Sorry, an error occurred while loading the content.

Re: "ocaml_beginners"::[] Best practice for closing network connections?

Expand Messages
  • rixed@happyleptic.org
    ... Can t you check this theory by listing established connections systemwide (what s netstat saying ?). What s after a while here ? After some time, or some
    Message 1 of 9 , Oct 13, 2009
    • 0 Attachment
      > I have a program that makes many many network connections, one after the other. After a while I am unable to make new connections, and I suspect it's because I'm not closing them properly.

      Can't you check this theory by listing established connections
      systemwide (what's netstat saying ?).

      What's "after a while" here ? After some time, or some number of
      established connections, or some amount of past connections ?

      Also, what's "unable to make new connections" mean ?
      You can use Unix.handle_unix_error to quickly find out which
      error you got.

      > let s = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
      > Unix.setsockopt_float s Unix.SO_RCVTIMEO 2.0;
      > Unix.setsockopt_float s Unix.SO_SNDTIMEO 2.0;
      > Unix.connect s address;
      > let inchan = Unix.in_channel_of_descr s in
      > let outchan = Unix.out_channel_of_descr s in
      >
      > To close the connection I do:
      >
      > Unix.shutdown s Unix.SHUTDOWN_SEND;
      > Unix.close s;

      As a side note, the shutdown seams useless since you close it
      afterward. Apart from that I can't see anything wrong neither.
    • Richard Jones
      ... SO_REUSEADDR? http://stackoverflow.com/questions/775638/using-soreuseaddr-what-happens-to-previously-open-socket/775657#775657 Rich. -- Richard Jones Red
      Message 2 of 9 , Oct 13, 2009
      • 0 Attachment
        On Tue, Oct 13, 2009 at 03:43:06AM -0000, jshaw10 wrote:
        > Hi Ocamlers,
        > I have a program that makes many many network connections, one after the other. After a while I am unable to make new connections, and I suspect it's because I'm not closing them properly.
        >
        > To create the connection I do:
        >
        > let s = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
        > Unix.setsockopt_float s Unix.SO_RCVTIMEO 2.0;
        > Unix.setsockopt_float s Unix.SO_SNDTIMEO 2.0;
        > Unix.connect s address;
        > let inchan = Unix.in_channel_of_descr s in
        > let outchan = Unix.out_channel_of_descr s in
        >
        > To close the connection I do:
        >
        > Unix.shutdown s Unix.SHUTDOWN_SEND;
        > Unix.close s;
        >
        > Am I missing anything?

        SO_REUSEADDR?

        http://stackoverflow.com/questions/775638/using-soreuseaddr-what-happens-to-previously-open-socket/775657#775657

        Rich.

        --
        Richard Jones
        Red Hat
      • jshaw10
        Rixed and Richard, thanks for your replies. Rixed, I used your idea and used TCPView to see how many connections I had open. Turned out I had a lot when some
        Message 3 of 9 , Oct 13, 2009
        • 0 Attachment
          Rixed and Richard, thanks for your replies. Rixed, I used your idea and used TCPView to see how many connections I had open. Turned out I had a lot when some sockets never connected to the server. So adding SO_REUSEADDR didn't help because I wasn't closing the connection after catching some exception or other.

          Anyway, that's mostly fixed. I ran the program and it crashed with 11 unclosed connections, which is much better. However, now one of the Unix.*channel_of_descr functions (the first one most likely) is giving me "Too many open files". I'll try using "close_in" and "close_out" before and after the Unix.close to see if that makes any difference.
        • jshaw10
          Interesting, now the program gives Fatal error: exception Sys_error( Bad file descriptor ) when it tries to load a file when it starts up.
          Message 4 of 9 , Oct 13, 2009
          • 0 Attachment
            Interesting, now the program gives Fatal error: exception Sys_error("Bad file descriptor") when it tries to load a file when it starts up.
          • jshaw10
            I changed the program to use Unix.write and Unix.read instead of using channels. Now it works fine.
            Message 5 of 9 , Oct 13, 2009
            • 0 Attachment
              I changed the program to use Unix.write and Unix.read instead of using channels. Now it works fine.
            • rixed@happyleptic.org
              -[ Tue, Oct 13, 2009 at 08:26:28PM -0000, jshaw10 ]---- ... You should check not only how many connections are established but also how many are still closing.
              Message 6 of 9 , Oct 14, 2009
              • 0 Attachment
                -[ Tue, Oct 13, 2009 at 08:26:28PM -0000, jshaw10 ]----
                > Rixed, I used your idea and used TCPView to see how many connections I had open.
                > Turned out I had a lot when some sockets never connected to the server.
                > So adding SO_REUSEADDR didn't help because I wasn't closing the connection after catching
                > some exception or other.

                You should check not only how many connections are established but also how
                many are still closing. I tried a programm that opens and close connections
                quickly (based on your code) and without REUSEADDR it fails after around 32K
                connections, as expected since it uses upper 32K ports for temporaryu local
                ports.

                > However, now one of the Unix.*channel_of_descr functions (the first one most
                > likely) is giving me "Too many open files". I'll try using "close_in" and
                > "close_out" before and after the Unix.close to see if that makes any
                > difference.

                As I understand it, these functions (Unix.*channel_of_descr) merely creates
                data structures around an already opened file descriptor, and are unlikelly
                to throw this "too many open files" exception. Are you sure this error is not
                raised by Unix.socket instead ?
              • Hugo Ferreira
                rixed@happyleptic.org wrote: ... May I suggest the original poster use: OCAMLRUNPARAM=b export OCAMLRUNPARAM to check where the exception actually occurs.
                Message 7 of 9 , Oct 14, 2009
                • 0 Attachment
                  rixed@... wrote:
                  ...
                  >> However, now one of the Unix.*channel_of_descr functions (the first one most
                  >> likely) is giving me "Too many open files". I'll try using "close_in" and
                  >> "close_out" before and after the Unix.close to see if that makes any
                  >> difference.
                  >
                  > As I understand it, these functions (Unix.*channel_of_descr) merely creates
                  > data structures around an already opened file descriptor, and are unlikelly
                  > to throw this "too many open files" exception. Are you sure this error is not
                  > raised by Unix.socket instead ?
                  >

                  May I suggest the original poster use:

                  OCAMLRUNPARAM=b
                  export OCAMLRUNPARAM

                  to check where the exception actually occurs.

                  regards,
                  HF


                  >
                  >
                  > ------------------------------------
                  >
                  > Archives up to December 31, 2008 are also downloadable at http://www.connettivo.net/cntprojects/ocaml_beginners/
                  > The archives of the very official ocaml list (the seniors' one) can be found at http://caml.inria.fr
                  > Attachments are banned and you're asked to be polite, avoid flames etc.Yahoo! Groups Links
                  >
                  >
                  >
                • jshaw10
                  I tried a programm that opens and close connections ... Thanks for trying that, since I didn t think to test the code with lots of connections in a short
                  Message 8 of 9 , Oct 14, 2009
                  • 0 Attachment
                    I tried a programm that opens and close connections
                    > quickly (based on your code) and without REUSEADDR it fails after around 32K
                    > connections, as expected since it uses upper 32K ports for temporaryu local
                    > ports.

                    Thanks for trying that, since I didn't think to test the code with lots of connections in a short period of time. In my program it runs around 5 minutes with say 800 connections.

                    > As I understand it, these functions (Unix.*channel_of_descr) merely creates
                    > data structures around an already opened file descriptor, and are unlikelly
                    > to throw this "too many open files" exception. Are you sure this error is not
                    > raised by Unix.socket instead ?

                    If I remember right the exception Unix_error reported that the offending function was channel_of_descr, which I suppose the *channel_of_descr functions use behind the scenes.

                    For some reason, Unix.read and Unix.write aren't causing me any errors. I think it's because it's much easier to handle errors. It seems I have fewer exceptions to catch. Using channels added an additional layer of exceptions such as Sys_error and Sys_blocked_io. Now I just check when Unix.read returns 0 and catch Unix_error. It may have been that I wasn't closing sockets properly in all cases.

                    My connection is good right now, so it would take some extra effort to test the robustness of my code, but I'm sure sometime today my wireless will become flaky again. Anyway, here's what I'm doing now:

                    let rec get_file accum rest =
                    let buff_size = 1024 in
                    match try Some (String.index rest '\n') with Not_found -> None with
                    Some i ->
                    get_file
                    ((String.sub rest 0 i)::accum)
                    (String.sub rest (i+1) (String.length rest - i - 1))
                    | None ->
                    let buffer = String.create buff_size in
                    let recv_n = Unix.read s buffer 0 buff_size in
                    if recv_n = 0
                    then List.rev accum
                    else get_file accum (rest ^ String.sub buffer 0 recv_n)

                    I call it with get_file [] "" and it returns a list of lines.
                  Your message has been successfully submitted and would be delivered to recipients shortly.