Loading ...
Sorry, an error occurred while loading the content.
 

Re: [nslu2-linux] Maximizing Slug Fileserver Speed

Expand Messages
  • Brian Wood
    ... These are SD files recorded by a MythTV system using PVR capture cards. They are about 2.2GB per hour (more or less full D1). The network is 100-base-T
    Message 1 of 6 , Jul 2 8:00 AM
      On Jul 2, 2006, at 6:25 AM, Yann E. MORIN wrote:

      > Brian,
      > All,
      >
      > On Sunday 02 July 2006 051, Brian Wood wrote:
      >> The slug will serve MPEG-2 video files, mostly over 1GB in size.
      >
      > How many at the same time? Which bitrate: 4Mibps (SDTV) or >8Mibps
      > (HDTV)?
      > What's between your slug and the player: WiFi, cat-5, PLT? One or
      > more hub
      > or switch? (see at bottom of mail).

      These are SD files recorded by a MythTV system using PVR capture
      cards. They are about 2.2GB per hour (more or less full D1). The
      network is 100-base-T hard-wired (CAT-5E) with one switch, not
      including the switch in the router.


      >
      >> I should use NFS and not Samba for maximum speed (not serving to
      >> Windows).
      >
      > To serve video (or music as well), why don't you use streaming?
      > That would have the benefit of adding no file system overhead on
      > the network, thus maximizing the bandwidth for the actual data
      > (provided your storage medium can sustain the throughput).

      I should have said that my purpose is not to "play" the files, but to
      give multiple machines access to the files in order edit them and
      create DVD images to burn.

      As such I don't think streaming would work, certainly not without
      major changes to the editing software. Throughput is not an issue the
      way it would be if playing, as there is no "real time" to keep up
      with, but obviously the faster the server the less time spent waiting
      while editing.


      >
      >> I should use a Big-Endian OS (not sure how much difference this might
      >> make).
      >
      >> From my experiments, that's not noticeable by timing with hardware,
      > and even less noticeable for a human being. That said, if you need
      > horsepower at the same time you transfer data, then you may loose
      > a few ticks, but that's nothing compared to any context-switch.

      There had been a few posts here recently that seemed to indicate that
      a TCP/IP-intensive machine would be more efficient if running BE, but
      I suspected the difference would be minor.

      But I'm tending to run the BE OpenSlug in any case, certainly I don't
      need the vast package library of the full Debian release just to run
      an NFS server.

      Unless, of course, somebody could convince me that OpenSlug is not
      the best answer here.


      >
      >> I should De-underclock the slug (seems obvious).
      >
      > That should help, but again, serving video via streaming is mostly
      > copy-from-disk-to-memory-to-network, and your limiting factors will
      > most
      > probably be (in path-order):
      > - disk-throughtput,
      > - IDE2USB adapter,
      > - USB throughput (you should be fine here!),
      > - memory latency,
      > - network throughput,
      > rather than CPU performance



      >
      >> Should I use a hard drive or a flash device for the OS ? (I'm
      >> assuming NFS would not give optimum speed).
      >
      > I don't get it right. Were your planing on an NFS-root? That
      > definitely
      > is a bad idea to maximise your bandwidth usage!
      > Anyway a hard drive for the rootfs would be better, you could even add
      > some swap to help in those moments where you'd need it.

      No NFS root, that would definitely not give me optimum speed. Just
      deciding between a hard drive and a flash device for root. I've never
      tried a flash stick, always used drives with nice large swap partitions.

      I've seen some "Fast Flash" NAND devices that are drop-in
      replacements for 2.5" hard drives. They claim 17.7MB/sec. read/write
      burst capability and 12MB sustained rates. Expensive, but look
      awfully nice for a root filesystem, and plenty of room for swap.
      Built-in "dynamic wear leveling", which I take to mean spreading out
      the load so you don't wear out specific locations quickly.


      >
      >> What filesystem should I use ?
      >
      > To store files, any _decent_ FS should do. ext2/3 or even reiserfs
      > should be
      > fine. Avoid FAT & co. and NTFS. Avoid any compressed FS (cramfs,
      > squashfs,
      > JFFS2...) because those _are_ CPU-bound.

      I think the first 3 you mentioned are the only simple choices for a
      slug in any case. I know the Myth people go on endlessly about what
      filesystem is best, usually with very conflicting results.


      Anyway, much thanks for your input, it is appreciated.
    • Brian Wood
      Apologies for not having found this earlier: http://www.nslu2-linux.org/wiki/Info/Performance Especially: Some tests done with the copy of a 200MB file between
      Message 2 of 6 , Jul 2 3:52 PM
        Apologies for not having found this earlier:

        http://www.nslu2-linux.org/wiki/Info/Performance

        Especially:

        Some tests done with the copy of a 200MB file between a NSLU2 with
        2.12-beta firmware and a Linux 2.8 (ubuntu), the test was performed
        with the filesystem "mounted" and then a simple read/write of the
        200MB via a python script.

        read write
        nfs: 5.7MB/s 2.6MB/s
        cifs: 3.5MB/s 1.9MB/s
        samba: 2.2MB/s 1.85MB/s

        nfs-server Version: 2.2beta47-2

        --titoo

        Looks like we're all doing a bit better than that, but that may also
        be with a 133Mhz. slug.
      Your message has been successfully submitted and would be delivered to recipients shortly.