Loading ...
Sorry, an error occurred while loading the content.

Re: Please evaluate my understanding wrt access files

Expand Messages
  • Noel Jones
    ... overhead is pretty much independent of file size. Each extra lookup table is another open file, buffer space, another library call (that might allocate
    Message 1 of 13 , Nov 2 5:45 AM
    • 0 Attachment
      On 10/31/2009 7:39 PM, Stan Hoeppner wrote:
      > Noel Jones put forth on 10/31/2009 1:12 AM:
      >> Each lookup table requires overhead. 30 separate tables requires
      >> considerably more overhead than one table. The size of the dataset
      >> doesn't change, it's the overhead that gets smaller. The more
      >> concurrent smtpd processes running, the more it matters.
      > I guess I was kinda looking for the technical definition of 'overhead'
      > here. If the overhead per file is a straight percentage, then it
      > shouldn't matter if it's one big file or 30 smaller files.

      "overhead" is pretty much independent of file size. Each
      extra lookup table is another open file, buffer space, another
      library call (that might allocate its own buffer), and an
      additional lookup. In a tiny environment it's not enough to
      make a difference -- probably not enough to even notice.

      But one lookup always beats two lookups, one open file always
      uses less resources than two open files. Run enough smtpd
      processes and it starts to make a difference; at some point it
      can make a critical difference. And it's always "prettier"
      to do one thing one time vs. doing the same thing over and over.

      > What is the
      > actual memory footprint overhead that's being added by Posfix for each
      > map file?

      Measure it on your system. Probably not a lot, but it adds up.

      > And is the amount of overhead the same for all map types, or
      > different/variable depending on map type?

      Yes, different table types use different system libraries and
      have different resource requirements. But for each table type
      it's a fairly constant value unaffected by table size (very
      small tables might use a little less RAM under some conditions
      -- but this really doesn't matter).

      > Is it something like 3 bytes
      > of overhead added for every 10 bytes in a map file, for example?

      No, it's fixed.

      > I can
      > understand that having everything in one map file, or few map files,
      > compared to many, can increase lookup performance due to indexing
      > efficiency. What I don't understand is how having fewer larger map
      > files decreases memory footprint.

      Fewer files = less overhead. Size doesn't matter here.

      > Also, given that cidr files aren't postmap'd, is it safe to assume they
      > aren't indexed? If they aren't indexed, how does putting the contents
      > in a single file speed up searching?

      One lookup beats two every time.

      > If they are indexed when Postfix
      > starts or reloads, then why wouldn't Postfix created a single index
      > _after_ reading in all the data from all the cidr files, creating one
      > internal program table?

      How is postfix supposed to know you want all these tables
      treated as one? Anyway, postfix would still need to open the
      files etc. so all you would save is the extra lookups. It's
      easy enough for you to control by building the tables the way
      you want them used that postfix shouldn't be guessing.

      -- Noel Jones
    Your message has been successfully submitted and would be delivered to recipients shortly.