Loading ...
Sorry, an error occurred while loading the content.

Re: Support for MDB in postfix 2.10?

Expand Messages
  • Wietse Venema
    ... I will evaluate the MDB client once there is a package for a mainstream LINUX or *BSD distribution that people can install by typing one or two commands.
    Message 1 of 20 , Feb 25 5:53 AM
    • 0 Attachment
      Patrick Lists:
      > On 02/25/2013 02:14 PM, Wietse Venema wrote:
      > > Quanah Gibson-Mount:
      > >>> I will evaluate the MDB client once there is an MDB library package
      > >>> for a main-stream Linux or *BSD platform.
      > >>>
      > >>> That is, something that installs MDB include files, an MDB object
      > >>> library file, and MDB documentation in /usr/{include,lib,man} or
      > >>> /usr/local/{include,lib,man}.
      > >>>
      > >>> You can accelerate this process by monitoring the repositories.
      > >>
      > >> Arch Linux now has a package for it:
      > >>
      > >> <https://aur.archlinux.org/packages/mdb-git/>
      > >
      > > This resolves to a totally-undocumented shell script with
      > > totally-undocumented external dependencies. No "usage" message, not
      > > even a manpage.
      >
      > Hi Wietse,
      >
      > The documentation for MDB can be found here:
      > http://symas.com/mdb/doc/index.html

      I will evaluate the MDB client once there is a package for a
      mainstream LINUX or *BSD distribution that people can install by
      typing one or two commands.

      Wietse
    • Patrick Lists
      On 02/25/2013 02:53 PM, Wietse Venema wrote: [snip] ... (Resend without the (S)RPMs since it exceeded the allowed size) Hi Wietse, It seems liblmdb is not yet
      Message 2 of 20 , Feb 25 9:02 AM
      • 0 Attachment
        On 02/25/2013 02:53 PM, Wietse Venema wrote:
        [snip]
        > I will evaluate the MDB client once there is a package for a
        > mainstream LINUX or *BSD distribution that people can install by
        > typing one or two commands.

        (Resend without the (S)RPMs since it exceeded the allowed size)

        Hi Wietse,

        It seems liblmdb is not yet available in Fedora, RHEL/CentOS. So I just
        created an RPM for RHEL6/CentOS6. Spec file attached. You can download
        the x86_64 RPMs & SRPM at http://pjl.home.xs4all.nl/downloads/liblmdb/

        Regards,
        Patrick
      • Wietse Venema
        ... FYI, in order to maintain Postfix in a meaningful manner I need to know what the library will look like on OTHER people s systems. Therefore I will wait
        Message 3 of 20 , Feb 25 10:19 AM
        • 0 Attachment
          Wietse:
          > I will evaluate the MDB client once there is a package for a
          > mainstream LINUX or *BSD distribution that people can install by
          > typing one or two commands.

          Patrick Lists:
          > It seems liblmdb is not yet available in Fedora, RHEL/CentOS. So I just
          > created an RPM for RHEL6/CentOS6. Spec file attached. You can download
          > the x86_64 RPMs & SRPM at http://pjl.home.xs4all.nl/downloads/liblmdb/

          FYI, in order to maintain Postfix in a meaningful manner I need to
          know what the library will look like on OTHER people's systems.
          Therefore I will wait until there is a package for a mainstream
          LINUX or *BSD distribution that OTHER people can install by typing
          one or two commands.

          Howard's Postfix client code from a few months ago requires mdb.h
          and libmdb.*. On the other hand your SRPM appears to install lmdb.h
          and liblmdb.*. Is this the beginning of a Babylonian confusion?
          What names will real systems use?

          Wietse
        • Patrick Lists
          ... Got it. I ll see what it takes to get this RPM included in Fedora and EPEL. If accepted that would cover Fedora, RHEL and CentOS. ... Judging from the git
          Message 4 of 20 , Feb 25 10:36 AM
          • 0 Attachment
            On 02/25/2013 07:19 PM, Wietse Venema wrote:
            > FYI, in order to maintain Postfix in a meaningful manner I need to
            > know what the library will look like on OTHER people's systems.
            > Therefore I will wait until there is a package for a mainstream
            > LINUX or *BSD distribution that OTHER people can install by typing
            > one or two commands.

            Got it. I'll see what it takes to get this RPM included in Fedora and
            EPEL. If accepted that would cover Fedora, RHEL and CentOS.

            > Howard's Postfix client code from a few months ago requires mdb.h
            > and libmdb.*. On the other hand your SRPM appears to install lmdb.h
            > and liblmdb.*. Is this the beginning of a Babylonian confusion?
            > What names will real systems use?

            Judging from the git commit log, on Nov 12, 2012 the "l" was added due
            to a naming conflict with other mdb* packages. See commit:

            http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13f3bcd59c2055d53e4759b0c3356b001eca44b0


            So the correct name is lmdb (liblmdb.so and lmdb.h).

            Regards,
            Patrick
          • Quanah Gibson-Mount
            --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists ... Specifically because there is a package on Debian that already provided libmdb.so for some
            Message 5 of 20 , Feb 25 11:10 AM
            • 0 Attachment
              --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists
              <postfix-list@...> wrote:

              > Judging from the git commit log, on Nov 12, 2012 the "l" was added due to
              > a naming conflict with other mdb* packages. See commit:
              >
              > http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13
              > f3bcd59c2055d53e4759b0c3356b001eca44b0
              >
              > So the correct name is lmdb (liblmdb.so and lmdb.h).

              Specifically because there is a package on Debian that already provided
              "libmdb.so" for some software from Microsoft. We couldn't find any
              conflict with liblmdb.so.

              --Quanah

              --

              Quanah Gibson-Mount
              Sr. Member of Technical Staff
              Zimbra, Inc
              A Division of VMware, Inc.
              --------------------
              Zimbra :: the leader in open source messaging and collaboration
            • Quanah Gibson-Mount
              --On Monday, February 25, 2013 11:10 AM -0800 Quanah Gibson-Mount ... Debian packaging issue: FreeBSD
              Message 6 of 20 , Feb 25 11:25 AM
              • 0 Attachment
                --On Monday, February 25, 2013 11:10 AM -0800 Quanah Gibson-Mount
                <quanah@...> wrote:

                > --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists
                > <postfix-list@...> wrote:
                >
                >> Judging from the git commit log, on Nov 12, 2012 the "l" was added due to
                >> a naming conflict with other mdb* packages. See commit:
                >>
                >> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13
                >> f3bcd59c2055d53e4759b0c3356b001eca44b0
                >>
                >> So the correct name is lmdb (liblmdb.so and lmdb.h).
                >
                > Specifically because there is a package on Debian that already provided
                > "libmdb.so" for some software from Microsoft. We couldn't find any
                > conflict with liblmdb.so.


                Debian packaging issue:
                <http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=694757>
                FreeBSD packaging issue: <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007>


                --Quanah


                --

                Quanah Gibson-Mount
                Sr. Member of Technical Staff
                Zimbra, Inc
                A Division of VMware, Inc.
                --------------------
                Zimbra :: the leader in open source messaging and collaboration
              • Wietse Venema
                ... I figured this was mdbtools (for Microsoft Access databases) which is available for Linux and BSD systems among others. Please keep me posted when there is
                Message 7 of 20 , Feb 25 11:29 AM
                • 0 Attachment
                  Quanah Gibson-Mount:
                  > --On Monday, February 25, 2013 7:36 PM +0100 Patrick Lists
                  > <postfix-list@...> wrote:
                  >
                  > > Judging from the git commit log, on Nov 12, 2012 the "l" was added due to
                  > > a naming conflict with other mdb* packages. See commit:
                  > >
                  > > http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commitdiff;h=13
                  > > f3bcd59c2055d53e4759b0c3356b001eca44b0
                  > >
                  > > So the correct name is lmdb (liblmdb.so and lmdb.h).
                  >
                  > Specifically because there is a package on Debian that already provided
                  > "libmdb.so" for some software from Microsoft. We couldn't find any
                  > conflict with liblmdb.so.

                  I figured this was mdbtools (for Microsoft Access databases) which
                  is available for Linux and BSD systems among others.

                  Please keep me posted when there is a package that *other* people
                  can install on a main-stream OS distribution, so that *I* know what
                  *their* Postfix will need work with.

                  Wietse
                • Quanah Gibson-Mount
                  --On Monday, February 25, 2013 2:29 PM -0500 Wietse Venema ... FreeBSD now has an official lmdb ports package.
                  Message 8 of 20 , Mar 11, 2013
                  • 0 Attachment
                    --On Monday, February 25, 2013 2:29 PM -0500 Wietse Venema
                    <wietse@...> wrote:

                    > Please keep me posted when there is a package that *other* people
                    > can install on a main-stream OS distribution, so that *I* know what
                    > *their* Postfix will need work with.

                    FreeBSD now has an official lmdb ports package.
                    <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007> has been closed.

                    --Quanah

                    --

                    Quanah Gibson-Mount
                    Sr. Member of Technical Staff
                    Zimbra, Inc
                    A Division of VMware, Inc.
                    --------------------
                    Zimbra :: the leader in open source messaging and collaboration
                  • Wietse Venema
                    ... I have updated the Postfix patch for lmdb databases. This is now included in snapshot 20130315. However, this code has several unexpected limitations. I
                    Message 9 of 20 , Mar 15, 2013
                    • 0 Attachment
                      Quanah Gibson-Mount:
                      > <wietse@...> wrote:
                      >
                      > > Please keep me posted when there is a package that *other* people
                      > > can install on a main-stream OS distribution, so that *I* know what
                      > > *their* Postfix will need work with.
                      >
                      > FreeBSD now has an official lmdb ports package.
                      > <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007> has been closed.

                      I have updated the Postfix patch for "lmdb" databases. This is now
                      included in snapshot 20130315.

                      However, this code has several unexpected limitations. I documented
                      the ones that I discovered today in LMDB_README. Because of the
                      limitations, the code is "snapshot only", i.e. it will not be part
                      of the stable release without major changes.

                      I suggest that people do not abandon CDB databases just yet.

                      Wietse
                    • Wietse Venema
                      ... Snaphot 20130317 addresses sub-optimal behavior in the LMDB client code that affected tlsmgr and postmap -i , and it makes the code more resilient. In
                      Message 10 of 20 , Mar 17, 2013
                      • 0 Attachment
                        Wietse Venema:
                        > Quanah Gibson-Mount:
                        > > <wietse@...> wrote:
                        > >
                        > > > Please keep me posted when there is a package that *other* people
                        > > > can install on a main-stream OS distribution, so that *I* know what
                        > > > *their* Postfix will need work with.
                        > >
                        > > FreeBSD now has an official lmdb ports package.
                        > > <http://www.freebsd.org/cgi/query-pr.cgi?pr=174007> has been closed.
                        >
                        > I have updated the Postfix patch for "lmdb" databases. This is now
                        > included in snapshot 20130315.

                        Snaphot 20130317 addresses sub-optimal behavior in the LMDB client
                        code that affected tlsmgr and "postmap -i", and it makes the code
                        more resilient.

                        In particular, the Postfix LMDB client will no longer keep crashing
                        on a "database full" error. Instead, Postfix can now recover without
                        immediately requiring human intervention. This is important because
                        many Postfix databases contain data that is maintained by a Postfix
                        daemon process, and the size of the data is not known in advance.

                        With this, LMDB no longer requires constant watching for "database
                        full" errors. As the system recovers from an error, it logs a warning
                        that humans can take care of the next day.

                        Wietse
                      • Quanah Gibson-Mount
                        --On Sunday, March 17, 2013 8:15 PM -0400 Wietse Venema ... Excellent news, thank you Wietse! Does this mean it may be included in a future release? --Quanah
                        Message 11 of 20 , Mar 18, 2013
                        • 0 Attachment
                          --On Sunday, March 17, 2013 8:15 PM -0400 Wietse Venema
                          <wietse@...> wrote:

                          > Snaphot 20130317 addresses sub-optimal behavior in the LMDB client
                          > code that affected tlsmgr and "postmap -i", and it makes the code
                          > more resilient.
                          >
                          > In particular, the Postfix LMDB client will no longer keep crashing
                          > on a "database full" error. Instead, Postfix can now recover without
                          > immediately requiring human intervention. This is important because
                          > many Postfix databases contain data that is maintained by a Postfix
                          > daemon process, and the size of the data is not known in advance.
                          >
                          > With this, LMDB no longer requires constant watching for "database
                          > full" errors. As the system recovers from an error, it logs a warning
                          > that humans can take care of the next day.

                          Excellent news, thank you Wietse! Does this mean it may be included in a
                          future release?

                          --Quanah


                          --

                          Quanah Gibson-Mount
                          Sr. Member of Technical Staff
                          Zimbra, Inc
                          A Division of VMware, Inc.
                          --------------------
                          Zimbra :: the leader in open source messaging and collaboration
                        • Quanah Gibson-Mount
                          --On Monday, March 18, 2013 4:26 PM -0400 Wietse Venema ... These sound like really excellent improvements. I ve been using MDB with OpenLDAP for a while, and
                          Message 12 of 20 , Mar 18, 2013
                          • 0 Attachment
                            --On Monday, March 18, 2013 4:26 PM -0400 Wietse Venema
                            <wietse@...> wrote:

                            > Quanah Gibson-Mount:
                            >> --On Sunday, March 17, 2013 8:15 PM -0400 Wietse Venema
                            >> <wietse@...> wrote:
                            >>
                            >> > Snaphot 20130317 addresses sub-optimal behavior in the LMDB client
                            >> > code that affected tlsmgr and "postmap -i", and it makes the code
                            >> > more resilient.
                            >> >
                            >> > In particular, the Postfix LMDB client will no longer keep crashing
                            >> > on a "database full" error. Instead, Postfix can now recover without
                            >> > immediately requiring human intervention. This is important because
                            >> > many Postfix databases contain data that is maintained by a Postfix
                            >> > daemon process, and the size of the data is not known in advance.
                            >> >
                            >> > With this, LMDB no longer requires constant watching for "database
                            >> > full" errors. As the system recovers from an error, it logs a warning
                            >> > that humans can take care of the next day.
                            >>
                            >> Excellent news, thank you Wietse! Does this mean it may be included in
                            >> a future release?
                            >
                            > Yes. And with these failure modes eliminated, it becomes worthwhile
                            > to explore new opportunities, such as updating a shared table.
                            >
                            > For example, multiple postscreen(8) or tlsmgr(8) or verify(8) daemons
                            > should be able to update a shared LMDB database, something that
                            > cannot safely be done with Berkeley DB. Occasionally some daemon
                            > may restart to automatically recover from an LMDB "table full"
                            > condition, but the system won't come to a halt. You can postpone
                            > these restarts by increasing lmdb_map_size in main.cf before the
                            > table reaches that limit.

                            These sound like really excellent improvements. I've been using MDB with
                            OpenLDAP for a while, and we're working hard on removing our reliance on
                            BDB everywhere, so now that postfix has support for it, it is a major win
                            in my book. Thank you!

                            --Quanah

                            --

                            Quanah Gibson-Mount
                            Sr. Member of Technical Staff
                            Zimbra, Inc
                            A Division of VMware, Inc.
                            --------------------
                            Zimbra :: the leader in open source messaging and collaboration
                          Your message has been successfully submitted and would be delivered to recipients shortly.