Loading ...
Sorry, an error occurred while loading the content.
 

Re: Support for MDB in postfix 2.10?

Expand Messages
  • Wietse Venema
    ... FYI, in order to maintain Postfix in a meaningful manner I need to know what the library will look like on OTHER people s systems. Therefore I will wait
    Message 1 of 20 , Feb 25, 2013
      Wietse:
      > I will evaluate the MDB client once there is a package for a
      > mainstream LINUX or *BSD distribution that people can install by
      > typing one or two commands.

      Patrick Lists:
      > It seems liblmdb is not yet available in Fedora, RHEL/CentOS. So I just
      > created an RPM for RHEL6/CentOS6. Spec file attached. You can download
      > the x86_64 RPMs & SRPM at http://pjl.home.xs4all.nl/downloads/liblmdb/

      FYI, in order to maintain Postfix in a meaningful manner I need to
      know what the library will look like on OTHER people's systems.
      Therefore I will wait until there is a package for a mainstream
      LINUX or *BSD distribution that OTHER people can install by typing
      one or two commands.

      Howard's Postfix client code from a few months ago requires mdb.h
      and libmdb.*. On the other hand your SRPM appears to install lmdb.h
      and liblmdb.*. Is this the beginning of a Babylonian confusion?
      What names will real systems use?

      Wietse
    • Patrick Lists
      ... Got it. I ll see what it takes to get this RPM included in Fedora and EPEL. If accepted that would cover Fedora, RHEL and CentOS. ... Judging from the git
      Message 2 of 20 , Feb 25, 2013
        On 02/25/2013 07:19 PM, Wietse Venema wrote:
        > FYI, in order to maintain Postfix in a meaningful manner I need to
        > know what the library will look like on OTHER people's systems.
        > Therefore I will wait until there is a package for a mainstream
        > LINUX or *BSD distribution that OTHER people can install by typing
        > one or two commands.

        Got it. I'll see what it takes to get this RPM included in Fedora and
        EPEL. If accepted that would cover Fedora, RHEL and CentOS.

        > Howard's Postfix client code from a few months ago requires mdb.h
        > and libmdb.*. On the other hand your SRPM appears to install lmdb.h
        > and liblmdb.*. Is this the beginning of a Babylonian confusion?
        > What names will real systems use?

        Judging from the git commit log, on Nov 12, 2012 the "l" was added due
        to a naming conflict with other mdb* packages. See commit:

        http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13f3bcd59c2055d53e4759b0c3356b001eca44b0


        So the correct name is lmdb (liblmdb.so and lmdb.h).

        Regards,
        Patrick
      • Quanah Gibson-Mount
        --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists ... Specifically because there is a package on Debian that already provided libmdb.so for some
        Message 3 of 20 , Feb 25, 2013
          --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists
          <postfix-list@...> wrote:

          > Judging from the git commit log, on Nov 12, 2012 the "l" was added due to
          > a naming conflict with other mdb* packages. See commit:
          >
          > http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13
          > f3bcd59c2055d53e4759b0c3356b001eca44b0
          >
          > So the correct name is lmdb (liblmdb.so and lmdb.h).

          Specifically because there is a package on Debian that already provided
          "libmdb.so" for some software from Microsoft. We couldn't find any
          conflict with liblmdb.so.

          --Quanah

          --

          Quanah Gibson-Mount
          Sr. Member of Technical Staff
          Zimbra, Inc
          A Division of VMware, Inc.
          --------------------
          Zimbra :: the leader in open source messaging and collaboration
        • Quanah Gibson-Mount
          --On Monday, February 25, 2013 11:10 AM -0800 Quanah Gibson-Mount ... Debian packaging issue: FreeBSD
          Message 4 of 20 , Feb 25, 2013
            --On Monday, February 25, 2013 11:10 AM -0800 Quanah Gibson-Mount
            <quanah@...> wrote:

            > --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists
            > <postfix-list@...> wrote:
            >
            >> Judging from the git commit log, on Nov 12, 2012 the "l" was added due to
            >> a naming conflict with other mdb* packages. See commit:
            >>
            >> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13
            >> f3bcd59c2055d53e4759b0c3356b001eca44b0
            >>
            >> So the correct name is lmdb (liblmdb.so and lmdb.h).
            >
            > Specifically because there is a package on Debian that already provided
            > "libmdb.so" for some software from Microsoft. We couldn't find any
            > conflict with liblmdb.so.


            Debian packaging issue:
            <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=694757>
            FreeBSD packaging issue: <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007>


            --Quanah


            --

            Quanah Gibson-Mount
            Sr. Member of Technical Staff
            Zimbra, Inc
            A Division of VMware, Inc.
            --------------------
            Zimbra :: the leader in open source messaging and collaboration
          • Wietse Venema
            ... I figured this was mdbtools (for Microsoft Access databases) which is available for Linux and BSD systems among others. Please keep me posted when there is
            Message 5 of 20 , Feb 25, 2013
              Quanah Gibson-Mount:
              > --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists
              > <postfix-list@...> wrote:
              >
              > > Judging from the git commit log, on Nov 12, 2012 the "l" was added due to
              > > a naming conflict with other mdb* packages. See commit:
              > >
              > > http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13
              > > f3bcd59c2055d53e4759b0c3356b001eca44b0
              > >
              > > So the correct name is lmdb (liblmdb.so and lmdb.h).
              >
              > Specifically because there is a package on Debian that already provided
              > "libmdb.so" for some software from Microsoft. We couldn't find any
              > conflict with liblmdb.so.

              I figured this was mdbtools (for Microsoft Access databases) which
              is available for Linux and BSD systems among others.

              Please keep me posted when there is a package that *other* people
              can install on a main-stream OS distribution, so that *I* know what
              *their* Postfix will need work with.

              Wietse
            • Quanah Gibson-Mount
              --On Monday, February 25, 2013 2:29 PM -0500 Wietse Venema ... FreeBSD now has an official lmdb ports package.
              Message 6 of 20 , Mar 11, 2013
                --On Monday, February 25, 2013 2:29 PM -0500 Wietse Venema
                <wietse@...> wrote:

                > Please keep me posted when there is a package that *other* people
                > can install on a main-stream OS distribution, so that *I* know what
                > *their* Postfix will need work with.

                FreeBSD now has an official lmdb ports package.
                <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007> has been closed.

                --Quanah

                --

                Quanah Gibson-Mount
                Sr. Member of Technical Staff
                Zimbra, Inc
                A Division of VMware, Inc.
                --------------------
                Zimbra :: the leader in open source messaging and collaboration
              • Wietse Venema
                ... I have updated the Postfix patch for lmdb databases. This is now included in snapshot 20130315. However, this code has several unexpected limitations. I
                Message 7 of 20 , Mar 15, 2013
                  Quanah Gibson-Mount:
                  > <wietse@...> wrote:
                  >
                  > > Please keep me posted when there is a package that *other* people
                  > > can install on a main-stream OS distribution, so that *I* know what
                  > > *their* Postfix will need work with.
                  >
                  > FreeBSD now has an official lmdb ports package.
                  > <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007> has been closed.

                  I have updated the Postfix patch for "lmdb" databases. This is now
                  included in snapshot 20130315.

                  However, this code has several unexpected limitations. I documented
                  the ones that I discovered today in LMDB_README. Because of the
                  limitations, the code is "snapshot only", i.e. it will not be part
                  of the stable release without major changes.

                  I suggest that people do not abandon CDB databases just yet.

                  Wietse
                • Wietse Venema
                  ... Snaphot 20130317 addresses sub-optimal behavior in the LMDB client code that affected tlsmgr and postmap -i , and it makes the code more resilient. In
                  Message 8 of 20 , Mar 17, 2013
                    Wietse Venema:
                    > Quanah Gibson-Mount:
                    > > <wietse@...> wrote:
                    > >
                    > > > Please keep me posted when there is a package that *other* people
                    > > > can install on a main-stream OS distribution, so that *I* know what
                    > > > *their* Postfix will need work with.
                    > >
                    > > FreeBSD now has an official lmdb ports package.
                    > > <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007> has been closed.
                    >
                    > I have updated the Postfix patch for "lmdb" databases. This is now
                    > included in snapshot 20130315.

                    Snaphot 20130317 addresses sub-optimal behavior in the LMDB client
                    code that affected tlsmgr and "postmap -i", and it makes the code
                    more resilient.

                    In particular, the Postfix LMDB client will no longer keep crashing
                    on a "database full" error. Instead, Postfix can now recover without
                    immediately requiring human intervention. This is important because
                    many Postfix databases contain data that is maintained by a Postfix
                    daemon process, and the size of the data is not known in advance.

                    With this, LMDB no longer requires constant watching for "database
                    full" errors. As the system recovers from an error, it logs a warning
                    that humans can take care of the next day.

                    Wietse
                  • Quanah Gibson-Mount
                    --On Sunday, March 17, 2013 8:15 PM -0400 Wietse Venema ... Excellent news, thank you Wietse! Does this mean it may be included in a future release? --Quanah
                    Message 9 of 20 , Mar 18, 2013
                      --On Sunday, March 17, 2013 8:15 PM -0400 Wietse Venema
                      <wietse@...> wrote:

                      > Snaphot 20130317 addresses sub-optimal behavior in the LMDB client
                      > code that affected tlsmgr and "postmap -i", and it makes the code
                      > more resilient.
                      >
                      > In particular, the Postfix LMDB client will no longer keep crashing
                      > on a "database full" error. Instead, Postfix can now recover without
                      > immediately requiring human intervention. This is important because
                      > many Postfix databases contain data that is maintained by a Postfix
                      > daemon process, and the size of the data is not known in advance.
                      >
                      > With this, LMDB no longer requires constant watching for "database
                      > full" errors. As the system recovers from an error, it logs a warning
                      > that humans can take care of the next day.

                      Excellent news, thank you Wietse! Does this mean it may be included in a
                      future release?

                      --Quanah


                      --

                      Quanah Gibson-Mount
                      Sr. Member of Technical Staff
                      Zimbra, Inc
                      A Division of VMware, Inc.
                      --------------------
                      Zimbra :: the leader in open source messaging and collaboration
                    • Quanah Gibson-Mount
                      --On Monday, March 18, 2013 4:26 PM -0400 Wietse Venema ... These sound like really excellent improvements. I ve been using MDB with OpenLDAP for a while, and
                      Message 10 of 20 , Mar 18, 2013
                        --On Monday, March 18, 2013 4:26 PM -0400 Wietse Venema
                        <wietse@...> wrote:

                        > Quanah Gibson-Mount:
                        >> --On Sunday, March 17, 2013 8:15 PM -0400 Wietse Venema
                        >> <wietse@...> wrote:
                        >>
                        >> > Snaphot 20130317 addresses sub-optimal behavior in the LMDB client
                        >> > code that affected tlsmgr and "postmap -i", and it makes the code
                        >> > more resilient.
                        >> >
                        >> > In particular, the Postfix LMDB client will no longer keep crashing
                        >> > on a "database full" error. Instead, Postfix can now recover without
                        >> > immediately requiring human intervention. This is important because
                        >> > many Postfix databases contain data that is maintained by a Postfix
                        >> > daemon process, and the size of the data is not known in advance.
                        >> >
                        >> > With this, LMDB no longer requires constant watching for "database
                        >> > full" errors. As the system recovers from an error, it logs a warning
                        >> > that humans can take care of the next day.
                        >>
                        >> Excellent news, thank you Wietse! Does this mean it may be included in
                        >> a future release?
                        >
                        > Yes. And with these failure modes eliminated, it becomes worthwhile
                        > to explore new opportunities, such as updating a shared table.
                        >
                        > For example, multiple postscreen(8) or tlsmgr(8) or verify(8) daemons
                        > should be able to update a shared LMDB database, something that
                        > cannot safely be done with Berkeley DB. Occasionally some daemon
                        > may restart to automatically recover from an LMDB "table full"
                        > condition, but the system won't come to a halt. You can postpone
                        > these restarts by increasing lmdb_map_size in main.cf before the
                        > table reaches that limit.

                        These sound like really excellent improvements. I've been using MDB with
                        OpenLDAP for a while, and we're working hard on removing our reliance on
                        BDB everywhere, so now that postfix has support for it, it is a major win
                        in my book. Thank you!

                        --Quanah

                        --

                        Quanah Gibson-Mount
                        Sr. Member of Technical Staff
                        Zimbra, Inc
                        A Division of VMware, Inc.
                        --------------------
                        Zimbra :: the leader in open source messaging and collaboration
                      Your message has been successfully submitted and would be delivered to recipients shortly.