Loading ...
Sorry, an error occurred while loading the content.

New "pipeline:" lookup table

Expand Messages
  • Wietse Venema
    This the second lookup table introduced with postfix-2.12-20140618. Wietse As the name suggests, the pipeline table implements a pipeline of lookup tables.
    Message 1 of 6 , Jun 18, 2014
    • 0 Attachment
      This the second lookup table introduced with postfix-2.12-20140618.

      Wietse

      As the name suggests, the "pipeline" table implements a pipeline
      of lookup tables. The name of the table specifies the pipeline as
      a sequence of tables. For example, the following prevents SMTP mail
      to system accounts that have "nologin" as their login shell:

      /etc/postfix/main.cf:
      local_recipient_maps =
      pipeline:!unix:passwd.byname!pcre:/etc/postfix/no-nologin.pcre
      alias_maps

      /etc/postfix/no-nologin.pcre:
      !/nologin/ whatever

      The ASCII character after "pipeline:" will be used as the separator
      between the lookup tables that follow (do not use space, ",", ":"
      or non-ASCII).

      Each "pipeline:" query is given to the first table. Each table
      lookup result becomes the query for the next table in the pipeline,
      and the last table produces the final result. When any table lookup
      produces no result, the entire pipeline produces no result.

      Some future version may support the form pipeline:/path/to/file,
      to load the list of lookup tables, one per line, from a textfile.
    • Jose Borges Ferreira
      ... That s great! Right now I can think a couple weird tricks i have to use on LDAP lookups that can be simplified with this. Can you consider a similar
      Message 2 of 6 , Jun 21, 2014
      • 0 Attachment
        On Wed, Jun 18, 2014 at 10:15 PM, Wietse Venema <wietse@...> wrote:
        > Each "pipeline:" query is given to the first table. Each table
        > lookup result becomes the query for the next table in the pipeline,
        > and the last table produces the final result. When any table lookup
        > produces no result, the entire pipeline produces no result.

        That's great!
        Right now I can think a couple weird tricks i have to use on LDAP
        lookups that can be simplified with this.

        Can you consider a similar behavior to allow caching.
        Something like:

        if ( value = get_table1 () ) {
        return value
        } else {
        value = get_table2();
        set_table1(value,ttl);
        return value
        }

        > Some future version may support the form pipeline:/path/to/file,
        > to load the list of lookup tables, one per line, from a textfile.

        Probably need that to allow my previous paragraph.

        José Borges Ferreira
      • Wietse Venema
        ... Why not implement a cache: maptype that you can prepend to the source (a pipeline any simple lookup table): cache:!maptype:mapname!ttl!pipeline... where
        Message 3 of 6 , Jun 21, 2014
        • 0 Attachment
          Jose Borges Ferreira:
          > On Wed, Jun 18, 2014 at 10:15 PM, Wietse Venema <wietse@...> wrote:
          > > Each "pipeline:" query is given to the first table. Each table
          > > lookup result becomes the query for the next table in the pipeline,
          > > and the last table produces the final result. When any table lookup
          > > produces no result, the entire pipeline produces no result.
          >
          > That's great!
          > Right now I can think a couple weird tricks i have to use on LDAP
          > lookups that can be simplified with this.
          >
          > Can you consider a similar behavior to allow caching.
          > Something like:
          >
          > if ( value = get_table1 () ) {
          > return value
          > } else {
          > value = get_table2();
          > set_table1(value,ttl);
          > return value
          > }

          Why not implement a cache: maptype that you can prepend to the
          source (a pipeline any simple lookup table):

          cache:!maptype:mapname!ttl!pipeline...

          where maptype:mapname implements persistent storage. This searches
          the persistent storage first, and if the item is not found or too
          old, invokes the source (pipeline or whatever) and stores a fresh
          result.

          The only problem is who invokes the cache cleanup operation.

          These pseudo tables are fun.

          Wietse
        • Jose Borges Ferreira
          ... Humm ... I usually go for memcache or redis for this kind o cache so I won t have to deal with expiration and cleanup. For lmdb,btree,mdb and other
          Message 4 of 6 , Jun 21, 2014
          • 0 Attachment
            On Sat, Jun 21, 2014 at 1:47 PM, Wietse Venema <wietse@...> wrote:
            > Jose Borges Ferreira:
            >> Can you consider a similar behavior to allow caching.
            >> Something like:
            >>
            >> if ( value = get_table1 () ) {
            >> return value
            >> } else {
            >> value = get_table2();
            >> set_table1(value,ttl);
            >> return value
            >> }
            >
            > Why not implement a cache: maptype that you can prepend to the
            > source (a pipeline any simple lookup table):
            >
            > cache:!maptype:mapname!ttl!pipeline...
            >
            > where maptype:mapname implements persistent storage. This searches
            > the persistent storage first, and if the item is not found or too
            > old, invokes the source (pipeline or whatever) and stores a fresh
            > result.
            >
            > The only problem is who invokes the cache cleanup operation.

            Humm ... I usually go for memcache or redis for this kind o cache so I
            won't have to deal with expiration and cleanup.
            For lmdb,btree,mdb and other persistent storage that is a pain.
            But caching is something that make sense at large scale and/or with
            slower backends and should be dealt with care. It not for everyday
            implementations.
            Have check internal table to see if it's a straightforward for this cases.

            José Borges Ferreira
          • Wietse Venema
            ... In that case the syntax would be: cache:!memcache:/etc/postfix/whatever!0:pipeline:.... There should be a source feature in the memcache client that is
            Message 5 of 6 , Jun 21, 2014
            • 0 Attachment
              Jose Borges Ferreira:
              > > Why not implement a cache: maptype that you can prepend to the
              > > source (a pipeline any simple lookup table):
              > >
              > > cache:!maptype:mapname!ttl!pipeline...
              > >
              > > where maptype:mapname implements persistent storage. This searches
              > > the persistent storage first, and if the item is not found or too
              > > old, invokes the source (pipeline or whatever) and stores a fresh
              > > result.
              > >
              > > The only problem is who invokes the cache cleanup operation.
              >
              > Humm ... I usually go for memcache or redis for this kind o cache so I
              > won't have to deal with expiration and cleanup.

              In that case the syntax would be:

              cache:!memcache:/etc/postfix/whatever!0:pipeline:....

              There should be a "source" feature in the memcache client that is
              like a read-only version of "backup". Then, one could say:

              /etc/postfix/main.cf:
              whatever = ... memcache:/etc/postfix/whatever ...

              /etc/postfix/whatever:
              source = pipeline:....

              The "source" feature would be perfect for DNS, reputation, etc.

              Wietse

              > For lmdb,btree,mdb and other persistent storage that is a pain.
              > But caching is something that make sense at large scale and/or with
              > slower backends and should be dealt with care. It not for everyday
              > implementations.
              > Have check internal table to see if it's a straightforward for this cases.
              >
              > Jos? Borges Ferreira
              >
            • Wietse Venema
              ... This is not needed. When a Postfix memcache is opened read-only, the backup database is accessed in read-only mode, and it is accesed only when information
              Message 6 of 6 , Jun 21, 2014
              • 0 Attachment
                Wietse Venema:
                > In that case the syntax would be:
                >
                > cache:!memcache:/etc/postfix/whatever!0:pipeline:....
                >
                > There should be a "source" feature in the memcache client that is
                > like a read-only version of "backup". Then, one could say:
                >
                > /etc/postfix/main.cf:
                > whatever = ... memcache:/etc/postfix/whatever ...
                >
                > /etc/postfix/whatever:
                > source = pipeline:....

                This is not needed. When a Postfix memcache is opened read-only,
                the backup database is accessed in read-only mode, and it is accesed
                only when information is not found in the cache.

                Thus:

                /etc/postfix/main.cf:
                whatever = ... memcache:/etc/postfix/whatever ...

                /etc/postfix/whatever:
                backup = (pipeline or table)

                should already work for read-only applications.

                Wietse
              Your message has been successfully submitted and would be delivered to recipients shortly.