Loading ...
Sorry, an error occurred while loading the content.

Re: "ocaml_beginners"::[] Best practice for closing network connections?

Expand Messages
  • Richard Jones
    ... SO_REUSEADDR? http://stackoverflow.com/questions/775638/using-soreuseaddr-what-happens-to-previously-open-socket/775657#775657 Rich. -- Richard Jones Red
    Message 1 of 9 , Oct 13, 2009
    • 0 Attachment
      On Tue, Oct 13, 2009 at 03:43:06AM -0000, jshaw10 wrote:
      > Hi Ocamlers,
      > I have a program that makes many many network connections, one after the other. After a while I am unable to make new connections, and I suspect it's because I'm not closing them properly.
      >
      > To create the connection I do:
      >
      > let s = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
      > Unix.setsockopt_float s Unix.SO_RCVTIMEO 2.0;
      > Unix.setsockopt_float s Unix.SO_SNDTIMEO 2.0;
      > Unix.connect s address;
      > let inchan = Unix.in_channel_of_descr s in
      > let outchan = Unix.out_channel_of_descr s in
      >
      > To close the connection I do:
      >
      > Unix.shutdown s Unix.SHUTDOWN_SEND;
      > Unix.close s;
      >
      > Am I missing anything?

      SO_REUSEADDR?

      http://stackoverflow.com/questions/775638/using-soreuseaddr-what-happens-to-previously-open-socket/775657#775657

      Rich.

      --
      Richard Jones
      Red Hat
    • jshaw10
      Rixed and Richard, thanks for your replies. Rixed, I used your idea and used TCPView to see how many connections I had open. Turned out I had a lot when some
      Message 2 of 9 , Oct 13, 2009
      • 0 Attachment
        Rixed and Richard, thanks for your replies. Rixed, I used your idea and used TCPView to see how many connections I had open. Turned out I had a lot when some sockets never connected to the server. So adding SO_REUSEADDR didn't help because I wasn't closing the connection after catching some exception or other.

        Anyway, that's mostly fixed. I ran the program and it crashed with 11 unclosed connections, which is much better. However, now one of the Unix.*channel_of_descr functions (the first one most likely) is giving me "Too many open files". I'll try using "close_in" and "close_out" before and after the Unix.close to see if that makes any difference.
      • jshaw10
        Interesting, now the program gives Fatal error: exception Sys_error( Bad file descriptor ) when it tries to load a file when it starts up.
        Message 3 of 9 , Oct 13, 2009
        • 0 Attachment
          Interesting, now the program gives Fatal error: exception Sys_error("Bad file descriptor") when it tries to load a file when it starts up.
        • jshaw10
          I changed the program to use Unix.write and Unix.read instead of using channels. Now it works fine.
          Message 4 of 9 , Oct 13, 2009
          • 0 Attachment
            I changed the program to use Unix.write and Unix.read instead of using channels. Now it works fine.
          • rixed@happyleptic.org
            -[ Tue, Oct 13, 2009 at 08:26:28PM -0000, jshaw10 ]---- ... You should check not only how many connections are established but also how many are still closing.
            Message 5 of 9 , Oct 14, 2009
            • 0 Attachment
              -[ Tue, Oct 13, 2009 at 08:26:28PM -0000, jshaw10 ]----
              > Rixed, I used your idea and used TCPView to see how many connections I had open.
              > Turned out I had a lot when some sockets never connected to the server.
              > So adding SO_REUSEADDR didn't help because I wasn't closing the connection after catching
              > some exception or other.

              You should check not only how many connections are established but also how
              many are still closing. I tried a programm that opens and close connections
              quickly (based on your code) and without REUSEADDR it fails after around 32K
              connections, as expected since it uses upper 32K ports for temporaryu local
              ports.

              > However, now one of the Unix.*channel_of_descr functions (the first one most
              > likely) is giving me "Too many open files". I'll try using "close_in" and
              > "close_out" before and after the Unix.close to see if that makes any
              > difference.

              As I understand it, these functions (Unix.*channel_of_descr) merely creates
              data structures around an already opened file descriptor, and are unlikelly
              to throw this "too many open files" exception. Are you sure this error is not
              raised by Unix.socket instead ?
            • Hugo Ferreira
              rixed@happyleptic.org wrote: ... May I suggest the original poster use: OCAMLRUNPARAM=b export OCAMLRUNPARAM to check where the exception actually occurs.
              Message 6 of 9 , Oct 14, 2009
              • 0 Attachment
                rixed@... wrote:
                ...
                >> However, now one of the Unix.*channel_of_descr functions (the first one most
                >> likely) is giving me "Too many open files". I'll try using "close_in" and
                >> "close_out" before and after the Unix.close to see if that makes any
                >> difference.
                >
                > As I understand it, these functions (Unix.*channel_of_descr) merely creates
                > data structures around an already opened file descriptor, and are unlikelly
                > to throw this "too many open files" exception. Are you sure this error is not
                > raised by Unix.socket instead ?
                >

                May I suggest the original poster use:

                OCAMLRUNPARAM=b
                export OCAMLRUNPARAM

                to check where the exception actually occurs.

                regards,
                HF


                >
                >
                > ------------------------------------
                >
                > Archives up to December 31, 2008 are also downloadable at http://www.connettivo.net/cntprojects/ocaml_beginners/
                > The archives of the very official ocaml list (the seniors' one) can be found at http://caml.inria.fr
                > Attachments are banned and you're asked to be polite, avoid flames etc.Yahoo! Groups Links
                >
                >
                >
              • jshaw10
                I tried a programm that opens and close connections ... Thanks for trying that, since I didn t think to test the code with lots of connections in a short
                Message 7 of 9 , Oct 14, 2009
                • 0 Attachment
                  I tried a programm that opens and close connections
                  > quickly (based on your code) and without REUSEADDR it fails after around 32K
                  > connections, as expected since it uses upper 32K ports for temporaryu local
                  > ports.

                  Thanks for trying that, since I didn't think to test the code with lots of connections in a short period of time. In my program it runs around 5 minutes with say 800 connections.

                  > As I understand it, these functions (Unix.*channel_of_descr) merely creates
                  > data structures around an already opened file descriptor, and are unlikelly
                  > to throw this "too many open files" exception. Are you sure this error is not
                  > raised by Unix.socket instead ?

                  If I remember right the exception Unix_error reported that the offending function was channel_of_descr, which I suppose the *channel_of_descr functions use behind the scenes.

                  For some reason, Unix.read and Unix.write aren't causing me any errors. I think it's because it's much easier to handle errors. It seems I have fewer exceptions to catch. Using channels added an additional layer of exceptions such as Sys_error and Sys_blocked_io. Now I just check when Unix.read returns 0 and catch Unix_error. It may have been that I wasn't closing sockets properly in all cases.

                  My connection is good right now, so it would take some extra effort to test the robustness of my code, but I'm sure sometime today my wireless will become flaky again. Anyway, here's what I'm doing now:

                  let rec get_file accum rest =
                  let buff_size = 1024 in
                  match try Some (String.index rest '\n') with Not_found -> None with
                  Some i ->
                  get_file
                  ((String.sub rest 0 i)::accum)
                  (String.sub rest (i+1) (String.length rest - i - 1))
                  | None ->
                  let buffer = String.create buff_size in
                  let recv_n = Unix.read s buffer 0 buff_size in
                  if recv_n = 0
                  then List.rev accum
                  else get_file accum (rest ^ String.sub buffer 0 recv_n)

                  I call it with get_file [] "" and it returns a list of lines.
                Your message has been successfully submitted and would be delivered to recipients shortly.