Loading ...
Sorry, an error occurred while loading the content.

New ftp server design.

Expand Messages
  • Beau Kuiper
    Hi all, I have been throwing a new anonymous ftp server design around in my head for the last couple of days. This design should easily be able to scale to
    Message 1 of 1 , Jul 21, 2000
    • 0 Attachment
      Hi all,

      I have been throwing a new anonymous ftp server design around in my head for the
      last couple of days. This design should easily be able to scale to many
      thousands of users. It will rely heavily on the mincore system call that is in
      2.4 kernels. It will do the following:

      1) Handle all data and control contections with one thread.
      2) Pre-cache the listing info of the entire ftp tree.
      3) Use memory mapping to transfer files and cache large amounts of data
      in memory.
      4) Use clone to create a thread for each device found in the ftp tree.
      Ie, if files are spread over 2 drives, create 2 threads. Since drives
      are access in serial, there is no point in more than one io thread per device
      5) When a disk read is forced (found out via mincore()), notify
      the thread that is connected to the device the file will be read from, and
      then the thread will do the io (access the data). The threads will also
      use mlock() to make sure the data stays memory.

      This architechure would require:

      1) Lots of memory. Start to swap, and everything goes to hell from
      there.
      2) A static document tree. This restriction could likely be removed at
      a later date.
      3) Does not scale really well with more than 1 processor. Again, could
      be fixed at a later date.

      However the benifits are:

      1) Extremely fast performace, even when document tree > ram size.
      2) No blocking of the main thread, due to disk reads being forced in
      other threads.
      3) list caching means no big hit when listing files.
      4) Fewer context switches than most other servers.
      5) When a server is blasted due to only a few files, this server
      should be able to handle it will due to data caching (no kernel calls
      to read it from disk again + cache is locked into memory)

      Are there any significant holes or problems with my idea that I haven't
      handled?

      BTW, I may never get around to fully implementing this, but i am about to have
      a go now, downloading 2.4.0test4

      Beau Kuiper
      kuiperba@...
    Your message has been successfully submitted and would be delivered to recipients shortly.