Loading ...
Sorry, an error occurred while loading the content.

Application API

Expand Messages
  • Nick Smith
    Hello Amanda Hackers, What is the status of the Application API in Amanda 2.6.1alpha? I ve compiled it on Solaris 10 because it would be great to use the said
    Message 1 of 23 , Aug 6, 2008
    • 0 Attachment
      Hello Amanda Hackers,

      What is the status of the Application API in Amanda 2.6.1alpha? I've
      compiled it on Solaris 10 because it would be great to use the said API
      to backup the ZFS volumes that I have. I have butchered the
      "amzfs-snapshot" script to work with the ZFS implementation of Solaris
      and it seems to work OK. There seems to be a lot of "commands" in said
      script that don't seem to be called and many the are "missing"
      ('estimate', 'backup' etc). Has the command naming convention changed
      since 2.6.0?

      Regards,

      Nick

      p.s. I will be more than happy to contribute any working scripts to the
      Amanda community.
    • Dustin J. Mitchell
      ... The Application API is actively under development, and has undergone some changes as we ve applied it and seen areas for improvement. Jean-Louis is doing
      Message 2 of 23 , Aug 6, 2008
      • 0 Attachment
        On Wed, Aug 6, 2008 at 9:12 AM, Nick Smith <nick.smith@...> wrote:
        > What is the status of the Application API in Amanda 2.6.1alpha? I've
        > compiled it on Solaris 10 because it would be great to use the said API to
        > backup the ZFS volumes that I have. I have butchered the "amzfs-snapshot"
        > script to work with the ZFS implementation of Solaris
        > and it seems to work OK. There seems to be a lot of "commands" in said
        > script that don't seem to be called and many the are "missing" ('estimate',
        > 'backup' etc). Has the command naming convention changed since 2.6.0?

        The Application API is actively under development, and has undergone
        some changes as we've applied it and seen areas for improvement.
        Jean-Louis is doing for most of the application API development while
        I concentrate on the transfer architecture and perl rewrite, so I can
        only give you this partial answer. He's away until 8/16 (lucky guy!).

        > p.s. I will be more than happy to contribute any working scripts to the
        > Amanda community.

        Great! Once Jean-Louis is back, I'm sure your feedback would be
        welcome -- an "outside" perspective will help us round out the API and
        smooth any rough edges.

        Dustin

        --
        Storage Software Engineer
        http://www.zmanda.com
      • Nick Smith
        Thanks Dustin, I have several questions / information for Jean-Louis when he returns from his vaction... I m interested in writing a qood quaility ZFS backup
        Message 3 of 23 , Aug 8, 2008
        • 0 Attachment
          Thanks Dustin,

          I have several questions / information for Jean-Louis when he returns
          from his vaction...

          I'm interested in writing a qood quaility ZFS backup application and I
          was wonder if there was anyone who was willing to review / discuss what
          the ZFS specific functionality? The first and probably most significant
          issue is how best to implement incremental backups (naming of snapshots
          etc etc).

          Regads,

          Nick

          Dustin J. Mitchell wrote:
          > On Wed, Aug 6, 2008 at 9:12 AM, Nick Smith <nick.smith@...> wrote:
          >> What is the status of the Application API in Amanda 2.6.1alpha? I've
          >> compiled it on Solaris 10 because it would be great to use the said API to
          >> backup the ZFS volumes that I have. I have butchered the "amzfs-snapshot"
          >> script to work with the ZFS implementation of Solaris
          >> and it seems to work OK. There seems to be a lot of "commands" in said
          >> script that don't seem to be called and many the are "missing" ('estimate',
          >> 'backup' etc). Has the command naming convention changed since 2.6.0?
          >
          > The Application API is actively under development, and has undergone
          > some changes as we've applied it and seen areas for improvement.
          > Jean-Louis is doing for most of the application API development while
          > I concentrate on the transfer architecture and perl rewrite, so I can
          > only give you this partial answer. He's away until 8/16 (lucky guy!).
          >
          >> p.s. I will be more than happy to contribute any working scripts to the
          >> Amanda community.
          >
          > Great! Once Jean-Louis is back, I'm sure your feedback would be
          > welcome -- an "outside" perspective will help us round out the API and
          > smooth any rough edges.
          >
          > Dustin
          >
        • Paddy Sreenivasan
          ... Nick, I think you are looking at amzfs-snapshot.pl in 2.6.1 tar ball. This script attempts to do backup from a zfs snapshot rather than directly from zfs
          Message 4 of 23 , Aug 8, 2008
          • 0 Attachment
            On Fri, Aug 8, 2008 at 1:51 AM, Nick Smith <nick.smith@...> wrote:
            > Thanks Dustin,
            >
            > I have several questions / information for Jean-Louis when he returns from
            > his vaction...
            >
            > I'm interested in writing a qood quaility ZFS backup application and I was
            > wonder if there was anyone who was willing to review / discuss what the ZFS
            > specific functionality? The first and probably most significant issue is how
            > best to implement incremental backups (naming of snapshots etc etc).

            Nick,

            I think you are looking at amzfs-snapshot.pl in 2.6.1 tar ball.

            This script attempts to do backup from a zfs snapshot rather than
            directly from zfs snapshot. So, as pre-backup, pre-estimate,
            pre-selfcheck
            action, it takes a snapshot and deletes in the post-action.

            If you have questions, please post them here.

            BTW, is there anyone who is interested in created a postgres backup
            application plugin?

            thanks,
            Paddy
            >
            > Regads,
            >
            > Nick
            >
            > Dustin J. Mitchell wrote:
            >>
            >> On Wed, Aug 6, 2008 at 9:12 AM, Nick Smith <nick.smith@...> wrote:
            >>>
            >>> What is the status of the Application API in Amanda 2.6.1alpha? I've
            >>> compiled it on Solaris 10 because it would be great to use the said API
            >>> to
            >>> backup the ZFS volumes that I have. I have butchered the "amzfs-snapshot"
            >>> script to work with the ZFS implementation of Solaris
            >>> and it seems to work OK. There seems to be a lot of "commands" in said
            >>> script that don't seem to be called and many the are "missing"
            >>> ('estimate',
            >>> 'backup' etc). Has the command naming convention changed since 2.6.0?
            >>
            >> The Application API is actively under development, and has undergone
            >> some changes as we've applied it and seen areas for improvement.
            >> Jean-Louis is doing for most of the application API development while
            >> I concentrate on the transfer architecture and perl rewrite, so I can
            >> only give you this partial answer. He's away until 8/16 (lucky guy!).
            >>
            >>> p.s. I will be more than happy to contribute any working scripts to the
            >>> Amanda community.
            >>
            >> Great! Once Jean-Louis is back, I'm sure your feedback would be
            >> welcome -- an "outside" perspective will help us round out the API and
            >> smooth any rough edges.
            >>
            >> Dustin
            >>
            >



            --
            Amanda http://amanda.zmanda.com
            ZRM for MySQL http://www.zmanda.com/backup-mysql.html
          • Jean-Louis Martineau
            ... I m back, As Paddy said, amzfs-snapshot is not an application, it is a script that run before and after the application. When executed as a pre script, it
            Message 5 of 23 , Aug 18, 2008
            • 0 Attachment
              Nick Smith wrote:
              > Thanks Dustin,
              >
              > I have several questions / information for Jean-Louis when he returns
              > from his vaction...

              I'm back,

              As Paddy said, amzfs-snapshot is not an application, it is a script that
              run before and after the application.

              When executed as a pre script, it create a snapshot, telling the
              application where the snapshot is mounted.
              When executed as a post script, it destroy the snapshot
              The amgtar application must be use to do the backup.

              You must define a dumptype that use the zfs-snapshot script and the
              amgtar application:

              #define application-tool for the amgtar application
              define application-tool app_amgtar {
              comment "amgtar"
              plugin "amgtar"
              #property "GNUTAR-PATH" "/path/to/gtar"
              #property "GNUTAR-LISTDIR" "/path/to/gnutar_list_dir"
              }

              #define script-tool for the amzfs-snapshot script
              define script-tool script_amzfs {
              comment "amzfs-snapshot"
              plugin "amzfs-snaphot"
              execute_where client
              execute_on
              pre-dle-amcheck,post-dle-amcheck,pre-dle-estimate,post-dle-estimate,pre-dle-backup,post-dle-backup
              #property "DF-PATH" "/path/to/df"
              #property "ZFS-PATH" "/path/to/zfs"
              }

              #define a dumptype that use amgtar and amzfs-snaphot
              define dumptype zfs {
              program "APPLICATION"
              application app_amgtar
              script script_amzfs
              }

              The DLE entry must use the zfs dumptype

              You must give amandabackup the priviledge to create and destroy snapshot
              with:
              zfs allow -ldu amandabackup mount,snapshot,destroy FILESYSTEM


              http://wiki.zmanda.com/index.php/Script_API
              http://wiki.zmanda.com/index.php/Application_API

              Jean-Louis
              >
              > I'm interested in writing a qood quaility ZFS backup application and I
              > was wonder if there was anyone who was willing to review / discuss
              > what the ZFS specific functionality? The first and probably most
              > significant issue is how best to implement incremental backups (naming
              > of snapshots etc etc).
              >
              > Regads,
              >
              > Nick
            • Nick Smith
              Hi Jean-Louis, The archiecture is a lot clearer now - many thanks. Comments & Questions (in no particular order) 1. Wouldn t zfs send be more
              Message 6 of 23 , Aug 19, 2008
              • 0 Attachment
                Hi Jean-Louis,

                The archiecture is a lot clearer now - many thanks.

                Comments & Questions (in no particular order)

                1. Wouldn't 'zfs send <snapshot> be more efficient than gtar (if
                indexing isn't required)? And how about incremental backups - surely ZFS
                would be faster?
                2. Does gtar support ZFS ACL's and properties?
                3. Having a pre and post script for ZFS is very useful - I will be
                using it to import and export ZFS pools that are on removable disks -
                I'll be writing an 'amzfs-importexport' script.
                4. The amzfs_snapshot script does not support ZFS volumes (unmounted
                raw virtual partitions). I use such partitions for applications such as
                Xen and generally backup the ZFS snapshots.
                5. Is there any reason why 'amzfs_snapshot' should not be
                simultaneously an application *and* a pre/post script?

                Regards,

                Nick

                Jean-Louis Martineau wrote:
                > Nick Smith wrote:
                >> Thanks Dustin,
                >>
                >> I have several questions / information for Jean-Louis when he returns
                >> from his vaction...
                >
                > I'm back,
                >
                > As Paddy said, amzfs-snapshot is not an application, it is a script that
                > run before and after the application.
                >
                > When executed as a pre script, it create a snapshot, telling the
                > application where the snapshot is mounted.
                > When executed as a post script, it destroy the snapshot
                > The amgtar application must be use to do the backup.
                >
                > You must define a dumptype that use the zfs-snapshot script and the
                > amgtar application:
                >
                > #define application-tool for the amgtar application
                > define application-tool app_amgtar {
                > comment "amgtar"
                > plugin "amgtar"
                > #property "GNUTAR-PATH" "/path/to/gtar"
                > #property "GNUTAR-LISTDIR" "/path/to/gnutar_list_dir"
                > }
                >
                > #define script-tool for the amzfs-snapshot script
                > define script-tool script_amzfs {
                > comment "amzfs-snapshot"
                > plugin "amzfs-snaphot"
                > execute_where client
                > execute_on
                >
                > pre-dle-amcheck,post-dle-amcheck,pre-dle-estimate,post-dle-estimate,pre-dle-backup,post-dle-backup
                >
                > #property "DF-PATH" "/path/to/df"
                > #property "ZFS-PATH" "/path/to/zfs"
                > }
                >
                > #define a dumptype that use amgtar and amzfs-snaphot
                > define dumptype zfs {
                > program "APPLICATION"
                > application app_amgtar
                > script script_amzfs
                > }
                >
                > The DLE entry must use the zfs dumptype
                >
                > You must give amandabackup the priviledge to create and destroy snapshot
                > with:
                > zfs allow -ldu amandabackup mount,snapshot,destroy FILESYSTEM
                >
                >
                > http://wiki.zmanda.com/index.php/Script_API
                > http://wiki.zmanda.com/index.php/Application_API
                >
                > Jean-Louis
                >>
                >> I'm interested in writing a qood quaility ZFS backup application and I
                >> was wonder if there was anyone who was willing to review / discuss
                >> what the ZFS specific functionality? The first and probably most
                >> significant issue is how best to implement incremental backups (naming
                >> of snapshots etc etc).
                >>
                >> Regads,
                >>
                >> Nick
                >
              • Jean-Louis Martineau
                ... zfs send should be faster but: - require more space in the pool, snapshot must be kept for the entire dumpcycle. - can t use include/exclude feature of
                Message 7 of 23 , Aug 19, 2008
                • 0 Attachment
                  Nick Smith wrote:
                  >
                  > Hi Jean-Louis,
                  >
                  > The archiecture is a lot clearer now - many thanks.
                  >
                  > Comments & Questions (in no particular order)
                  >
                  > 1. Wouldn't 'zfs send <snapshot> be more efficient than gtar (if
                  > indexing isn't required)? And how about incremental backups - surely
                  > ZFS would be faster?
                  'zfs send' should be faster but:
                  - require more space in the pool, snapshot must be kept for the entire
                  dumpcycle.
                  - can't use include/exclude feature of amgtar
                  - I don't know if it is possible to generate an index (-v option)
                  - you can't restore a single file, you must restore the complete stream.
                  > 2. Does gtar support ZFS ACL's and properties?
                  Need to be tested.
                  > 3. Having a pre and post script for ZFS is very useful - I will be
                  > using it to import and export ZFS pools that are on removable disks -
                  > I'll be writing an 'amzfs-importexport' script.
                  > 4. The amzfs_snapshot script does not support ZFS volumes (unmounted
                  > raw virtual partitions). I use such partitions for applications such
                  > as Xen and generally backup the ZFS snapshots.
                  amgtar require the volume to be mounted
                  > 5. Is there any reason why 'amzfs_snapshot' should not be
                  > simultaneously an application *and* a pre/post script?
                  No, if you do something with 'zfs send', it must be an application.

                  Jean-Louis
                • Nick Smith
                  Hi Jean-Louis, I ve been testing using your suggested configuration and I m getting the error from amdump : driver: WARNING: got empty schedule from planner
                  Message 8 of 23 , Sep 15, 2008
                  • 0 Attachment
                    Hi Jean-Louis,

                    I've been testing using your suggested configuration and I'm getting the
                    error from amdump : 'driver: WARNING: got empty schedule from planner'
                    when I define the script_amzfs to execute on the client. From the
                    debugging that I've done it seems that the planner just skips the client
                    side script processing.

                    How far is the script functionality? Should I be waiting until 2.6.1b1
                    to start testing?

                    Enhancement Request :

                    Would it be possible to define two more 'execute_on' hooks;
                    'pre-dumprun' & 'post-dumprun'? These hooks would be useful for tasks
                    such as importing and exporting Hot-Swap ZFS volumes for offsite
                    hard-disk backups. I know that I could do this as part of the 'cron'
                    script or in a wrapper script but it would be cleaner and more
                    functional if integrated into Amanda.

                    Regards & Many Thanks!

                    Nick

                    Jean-Louis Martineau wrote:
                    > Nick Smith wrote:
                    >> Thanks Dustin,
                    >>
                    >> I have several questions / information for Jean-Louis when he returns
                    >> from his vaction...
                    >
                    > I'm back,
                    >
                    > As Paddy said, amzfs-snapshot is not an application, it is a script that
                    > run before and after the application.
                    >
                    > When executed as a pre script, it create a snapshot, telling the
                    > application where the snapshot is mounted.
                    > When executed as a post script, it destroy the snapshot
                    > The amgtar application must be use to do the backup.
                    >
                    > You must define a dumptype that use the zfs-snapshot script and the
                    > amgtar application:
                    >
                    > #define application-tool for the amgtar application
                    > define application-tool app_amgtar {
                    > comment "amgtar"
                    > plugin "amgtar"
                    > #property "GNUTAR-PATH" "/path/to/gtar"
                    > #property "GNUTAR-LISTDIR" "/path/to/gnutar_list_dir"
                    > }
                    >
                    > #define script-tool for the amzfs-snapshot script
                    > define script-tool script_amzfs {
                    > comment "amzfs-snapshot"
                    > plugin "amzfs-snaphot"
                    > execute_where client
                    > execute_on
                    >
                    > pre-dle-amcheck,post-dle-amcheck,pre-dle-estimate,post-dle-estimate,pre-dle-backup,post-dle-backup
                    >
                    > #property "DF-PATH" "/path/to/df"
                    > #property "ZFS-PATH" "/path/to/zfs"
                    > }
                    >
                    > #define a dumptype that use amgtar and amzfs-snaphot
                    > define dumptype zfs {
                    > program "APPLICATION"
                    > application app_amgtar
                    > script script_amzfs
                    > }
                    >
                    > The DLE entry must use the zfs dumptype
                    >
                    > You must give amandabackup the priviledge to create and destroy snapshot
                    > with:
                    > zfs allow -ldu amandabackup mount,snapshot,destroy FILESYSTEM
                    >
                    >
                    > http://wiki.zmanda.com/index.php/Script_API
                    > http://wiki.zmanda.com/index.php/Application_API
                    >
                    > Jean-Louis
                    >>
                    >> I'm interested in writing a qood quaility ZFS backup application and I
                    >> was wonder if there was anyone who was willing to review / discuss
                    >> what the ZFS specific functionality? The first and probably most
                    >> significant issue is how best to implement incremental backups (naming
                    >> of snapshots etc etc).
                    >>
                    >> Regads,
                    >>
                    >> Nick
                    >
                  • Jean-Louis Martineau
                    ... Script works, I have done only small testing. Can you send me the debug files and the output of amadmin disklist I want the amandad.*.debug,
                    Message 9 of 23 , Sep 15, 2008
                    • 0 Attachment
                      Nick Smith wrote:
                      > Hi Jean-Louis,
                      >
                      > I've been testing using your suggested configuration and I'm getting the
                      > error from amdump : 'driver: WARNING: got empty schedule from planner'
                      > when I define the script_amzfs to execute on the client. From the
                      > debugging that I've done it seems that the planner just skips the
                      > client side script processing.
                      >
                      > How far is the script functionality? Should I be waiting until 2.6.1b1
                      > to start testing?
                      Script works, I have done only small testing.
                      Can you send me the debug files and the output of "amadmin <CONFIG>
                      disklist"
                      I want the amandad.*.debug, selfcheck.*.debug, sendsize.*.debug and the
                      the zfs*.*.debug files

                      >
                      > Enhancement Request :
                      >
                      > Would it be possible to define two more 'execute_on' hooks;
                      > 'pre-dumprun' & 'post-dumprun'? These hooks would be useful for tasks
                      > such as importing and exporting Hot-Swap ZFS volumes for offsite
                      > hard-disk backups. I know that I could do this as part of the 'cron'
                      > script or in a wrapper script but it would be cleaner and more
                      > functional if integrated into Amanda.
                      Use pre-host-estimate and post-host-backup.

                      Jean-Louis
                    • Jean-Louis Martineau
                      Nick, It looks like the server process are crashing but I can t reproduce it even if I use your config. Can you add debug_auth 1 and debug_protocol 1 in
                      Message 10 of 23 , Sep 16, 2008
                      • 0 Attachment
                        Nick,

                        It looks like the server process are crashing but I can't reproduce it
                        even if I use your config.

                        Can you add "debug_auth 1" and "debug_protocol 1" in your amanda.conf
                        And run: amcheck -c daily
                        Send me the output of the program and the amcheck debug file.

                        Jean-Louis

                        Nick Smith wrote:
                        > Hi Jena-Louis,
                        >
                        > Please ignore my previous confused email. Here is some trace with the
                        > 'execute_on' property set to 'client'!
                        >
                        > Regards,
                        >
                        > Nick
                      • Nick Smith
                        Jean-Louis, Here is the amcheck debug file. Regards, Nick FYI : I m running on Solaris 10 with Amanda built using the GLIB from supplied with the OS and Sun
                        Message 11 of 23 , Sep 16, 2008
                        • 0 Attachment
                          Jean-Louis,

                          Here is the amcheck debug file.

                          Regards,

                          Nick

                          FYI : I'm running on Solaris 10 with Amanda built using the GLIB from
                          supplied with the OS and Sun Studio 12.

                          Jean-Louis Martineau wrote:
                          > Nick,
                          >
                          > It looks like the server process are crashing but I can't reproduce it
                          > even if I use your config.
                          >
                          > Can you add "debug_auth 1" and "debug_protocol 1" in your amanda.conf
                          > And run: amcheck -c daily
                          > Send me the output of the program and the amcheck debug file.
                          >
                          > Jean-Louis
                          >
                          > Nick Smith wrote:
                          >> Hi Jena-Louis,
                          >>
                          >> Please ignore my previous confused email. Here is some trace with the
                          >> 'execute_on' property set to 'client'!
                          >>
                          >> Regards,
                          >>
                          >> Nick
                        • Jean-Louis Martineau
                          Nick, I haven t found where it crash, can you apply the attached patch and retry amcheck -c daily ? Send me the amcheck debug file. Jean-Louis
                          Message 12 of 23 , Sep 17, 2008
                          • 0 Attachment
                            Nick,

                            I haven't found where it crash, can you apply the attached patch and
                            retry 'amcheck -c daily'?
                            Send me the amcheck debug file.

                            Jean-Louis

                            Nick Smith wrote:
                            > Pardon,
                            >
                            > Forgot to hit the 'attach' button.
                            >
                            > Sorry!
                            >
                            > Nick
                            >
                            >
                            > Jean-Louis Martineau wrote:
                            >> Where is it?
                            >>
                            >> Jean-Louis
                            >>
                            >> Nick Smith wrote:
                            >>> Jean-Louis,
                            >>>
                            >>> Here is the amcheck debug file.
                            >>>
                            >>> Regards,
                            >>>
                            >>> Nick
                            >>>
                            >>> FYI : I'm running on Solaris 10 with Amanda built using the GLIB
                            >>> from supplied with the OS and Sun Studio 12.
                            >>>
                            >>> Jean-Louis Martineau wrote:
                            >>>> Nick,
                            >>>>
                            >>>> It looks like the server process are crashing but I can't reproduce
                            >>>> it even if I use your config.
                            >>>>
                            >>>> Can you add "debug_auth 1" and "debug_protocol 1" in your amanda.conf
                            >>>> And run: amcheck -c daily
                            >>>> Send me the output of the program and the amcheck debug file.
                            >>>>
                            >>>> Jean-Louis
                            >>>>
                            >>>> Nick Smith wrote:
                            >>>>> Hi Jena-Louis,
                            >>>>>
                            >>>>> Please ignore my previous confused email. Here is some trace with
                            >>>>> the 'execute_on' property set to 'client'!
                            >>>>>
                            >>>>> Regards,
                            >>>>>
                            >>>>> Nick
                            >>
                          • Nick Smith
                            Hi Jean-Louis, Many thanks for the patch - it s helped a great deal. I ve been through a few cycles of adding my own debug and have found the crash problem (a
                            Message 13 of 23 , Sep 18, 2008
                            • 0 Attachment
                              Hi Jean-Louis,

                              Many thanks for the patch - it's helped a great deal.

                              I've been through a few cycles of adding my own debug and have found the
                              crash problem (a missing NULL on a varags parameter - see patch).

                              Now the amzfs-snapshot script runs correctly and the backup succeeds!

                              Again, many thanks for your assistance!

                              Nick

                              Jean-Louis Martineau wrote:
                              > Nick,
                              >
                              > I haven't found where it crash, can you apply the attached patch and
                              > retry 'amcheck -c daily'?
                              > Send me the amcheck debug file.
                              >
                              > Jean-Louis
                              >
                              > Nick Smith wrote:
                              >> Pardon,
                              >>
                              >> Forgot to hit the 'attach' button.
                              >>
                              >> Sorry!
                              >>
                              >> Nick
                              >>
                              >>
                              >> Jean-Louis Martineau wrote:
                              >>> Where is it?
                              >>>
                              >>> Jean-Louis
                              >>>
                              >>> Nick Smith wrote:
                              >>>> Jean-Louis,
                              >>>>
                              >>>> Here is the amcheck debug file.
                              >>>>
                              >>>> Regards,
                              >>>>
                              >>>> Nick
                              >>>>
                              >>>> FYI : I'm running on Solaris 10 with Amanda built using the GLIB
                              >>>> from supplied with the OS and Sun Studio 12.
                              >>>>
                              >>>> Jean-Louis Martineau wrote:
                              >>>>> Nick,
                              >>>>>
                              >>>>> It looks like the server process are crashing but I can't reproduce
                              >>>>> it even if I use your config.
                              >>>>>
                              >>>>> Can you add "debug_auth 1" and "debug_protocol 1" in your amanda.conf
                              >>>>> And run: amcheck -c daily
                              >>>>> Send me the output of the program and the amcheck debug file.
                              >>>>>
                              >>>>> Jean-Louis
                              >>>>>
                              >>>>> Nick Smith wrote:
                              >>>>>> Hi Jena-Louis,
                              >>>>>>
                              >>>>>> Please ignore my previous confused email. Here is some trace with
                              >>>>>> the 'execute_on' property set to 'client'!
                              >>>>>>
                              >>>>>> Regards,
                              >>>>>>
                              >>>>>> Nick
                              >>>
                              >
                            • Jean-Louis Martineau
                              Nick, Thanks for your help to find this bug. Jean-Louis
                              Message 14 of 23 , Sep 18, 2008
                              • 0 Attachment
                                Nick,

                                Thanks for your help to find this bug.

                                Jean-Louis

                                Nick Smith wrote:
                                > Hi Jean-Louis,
                                >
                                > Many thanks for the patch - it's helped a great deal.
                                >
                                > I've been through a few cycles of adding my own debug and have found
                                > the crash problem (a missing NULL on a varags parameter - see patch).
                                >
                                > Now the amzfs-snapshot script runs correctly and the backup succeeds!
                                >
                                > Again, many thanks for your assistance!
                                >
                                > Nick
                              • Nick Smith
                                Hi Jean-Louis, No problem. Should I file a bug somewhere - can t find any links to the bug database? Also there are some typo s in the example amanda.conf
                                Message 15 of 23 , Sep 18, 2008
                                • 0 Attachment
                                  Hi Jean-Louis,

                                  No problem. Should I file a bug somewhere - can't find any links to the
                                  bug database?

                                  Also there are some typo's in the example amanda.conf where it defines
                                  the 'sc-email' script tool. Some of the 'execute_on' hooks have
                                  obviously been renamed from 'pre-dle-selfcheck' to 'pre-dle-amcheck' etc.

                                  Regards,

                                  Nick


                                  Jean-Louis Martineau wrote:
                                  > Nick,
                                  >
                                  > Thanks for your help to find this bug.
                                  >
                                  > Jean-Louis
                                  >
                                  > Nick Smith wrote:
                                  >> Hi Jean-Louis,
                                  >>
                                  >> Many thanks for the patch - it's helped a great deal.
                                  >>
                                  >> I've been through a few cycles of adding my own debug and have found
                                  >> the crash problem (a missing NULL on a varags parameter - see patch).
                                  >>
                                  >> Now the amzfs-snapshot script runs correctly and the backup succeeds!
                                  >>
                                  >> Again, many thanks for your assistance!
                                  >>
                                  >> Nick
                                  >
                                • Dustin J. Mitchell
                                  ... We re not currently using the SF bug tracker -- it s difficult to use efficiently. Jean-Louis has already posted a patch for review, and we ll get it
                                  Message 16 of 23 , Sep 18, 2008
                                  • 0 Attachment
                                    On Thu, Sep 18, 2008 at 11:03 AM, Nick Smith <nick.smith@...> wrote:
                                    > No problem. Should I file a bug somewhere - can't find any links to the bug
                                    > database?

                                    We're not currently using the SF bug tracker -- it's difficult to use
                                    efficiently. Jean-Louis has already posted a patch for review, and
                                    we'll get it committed (with credit to you) shortly. In general, our
                                    work on bugs is sufficiently low-volume that this list is effective.
                                    If that starts to scale up -- and I hope it does! -- then we'll look
                                    for alternatives.

                                    > Also there are some typo's in the example amanda.conf where it defines the
                                    > 'sc-email' script tool. Some of the 'execute_on' hooks have obviously been
                                    > renamed from 'pre-dle-selfcheck' to 'pre-dle-amcheck' etc.

                                    A patch would be wonderful!

                                    Dustin

                                    --
                                    Storage Software Engineer
                                    http://www.zmanda.com
                                  • Paddy Sreenivasan
                                    ... Fix is available in Amanda sourceforge trunk. Paddy ... -- Amanda http://amanda.zmanda.com ZRM for MySQL http://www.zmanda.com/backup-mysql.html
                                    Message 17 of 23 , Sep 18, 2008
                                    • 0 Attachment
                                      On Thu, Sep 18, 2008 at 10:41 AM, Dustin J. Mitchell <dustin@...> wrote:
                                      > On Thu, Sep 18, 2008 at 11:03 AM, Nick Smith <nick.smith@...> wrote:
                                      >> No problem. Should I file a bug somewhere - can't find any links to the bug
                                      >> database?
                                      >
                                      > We're not currently using the SF bug tracker -- it's difficult to use
                                      > efficiently. Jean-Louis has already posted a patch for review, and
                                      > we'll get it committed (with credit to you) shortly. In general, our
                                      > work on bugs is sufficiently low-volume that this list is effective.
                                      > If that starts to scale up -- and I hope it does! -- then we'll look
                                      > for alternatives.

                                      Fix is available in Amanda sourceforge trunk.

                                      Paddy
                                      >
                                      >> Also there are some typo's in the example amanda.conf where it defines the
                                      >> 'sc-email' script tool. Some of the 'execute_on' hooks have obviously been
                                      >> renamed from 'pre-dle-selfcheck' to 'pre-dle-amcheck' etc.
                                      >
                                      > A patch would be wonderful!
                                      >
                                      > Dustin
                                      >
                                      > --
                                      > Storage Software Engineer
                                      > http://www.zmanda.com
                                      >



                                      --
                                      Amanda http://amanda.zmanda.com
                                      ZRM for MySQL http://www.zmanda.com/backup-mysql.html
                                    • Nick Smith
                                      ... Understood. As a developer I agree that bug databases don t make sense unless there is a substantial number of engineers working on the code base and also
                                      Message 18 of 23 , Sep 19, 2008
                                      • 0 Attachment
                                        Dustin J. Mitchell wrote:
                                        > On Thu, Sep 18, 2008 at 11:03 AM, Nick Smith <nick.smith@...> wrote:
                                        >> No problem. Should I file a bug somewhere - can't find any links to the bug
                                        >> database?
                                        >
                                        > We're not currently using the SF bug tracker -- it's difficult to use
                                        > efficiently. Jean-Louis has already posted a patch for review, and
                                        > we'll get it committed (with credit to you) shortly. In general, our
                                        > work on bugs is sufficiently low-volume that this list is effective.
                                        > If that starts to scale up -- and I hope it does! -- then we'll look
                                        > for alternatives.
                                        >
                                        Understood. As a developer I agree that bug databases don't make sense
                                        unless there is a substantial number of engineers working on the code
                                        base and also a good number of bug reports!

                                        >> Also there are some typo's in the example amanda.conf where it defines the
                                        >> 'sc-email' script tool. Some of the 'execute_on' hooks have obviously been
                                        >> renamed from 'pre-dle-selfcheck' to 'pre-dle-amcheck' etc.
                                        >
                                        > A patch would be wonderful!
                                        >
                                        Attached! :-)

                                        > Dustin
                                        >

                                        Regards,

                                        Nick
                                      • Paddy Sreenivasan
                                        On Tue, Aug 19, 2008 at 6:19 AM, Jean-Louis Martineau ... Nick, are you working on an application plugin that uses zfs send/recv instead of gtar? thanks, Paddy
                                        Message 19 of 23 , Oct 1, 2008
                                        • 0 Attachment
                                          On Tue, Aug 19, 2008 at 6:19 AM, Jean-Louis Martineau
                                          <martineau@...> wrote:
                                          > Nick Smith wrote:
                                          >>
                                          >> Hi Jean-Louis,
                                          >>
                                          >> The archiecture is a lot clearer now - many thanks.
                                          >>
                                          >> Comments & Questions (in no particular order)
                                          >>
                                          >> 1. Wouldn't 'zfs send <snapshot> be more efficient than gtar (if indexing
                                          >> isn't required)? And how about incremental backups - surely ZFS would be
                                          >> faster?
                                          >
                                          > 'zfs send' should be faster but:
                                          > - require more space in the pool, snapshot must be kept for the entire
                                          > dumpcycle.
                                          > - can't use include/exclude feature of amgtar
                                          > - I don't know if it is possible to generate an index (-v option)
                                          > - you can't restore a single file, you must restore the complete stream.
                                          >>

                                          Nick, are you working on an application plugin that uses zfs send/recv
                                          instead of gtar?

                                          thanks,
                                          Paddy
                                        • Nick Smith
                                          ... Hi Paddy, Yes I am. I got a zfs send/recv implementation working with 2.6.0 but I lost it during the upgrade to 2.6.1 (idiot). The follow on application is
                                          Message 20 of 23 , Oct 2, 2008
                                          • 0 Attachment
                                            Paddy Sreenivasan wrote:
                                            > On Tue, Aug 19, 2008 at 6:19 AM, Jean-Louis Martineau
                                            > <martineau@...> wrote:
                                            >> Nick Smith wrote:
                                            >>> Hi Jean-Louis,
                                            >>>
                                            >>> The archiecture is a lot clearer now - many thanks.
                                            >>>
                                            >>> Comments & Questions (in no particular order)
                                            >>>
                                            >>> 1. Wouldn't 'zfs send <snapshot> be more efficient than gtar (if indexing
                                            >>> isn't required)? And how about incremental backups - surely ZFS would be
                                            >>> faster?
                                            >> 'zfs send' should be faster but:
                                            >> - require more space in the pool, snapshot must be kept for the entire
                                            >> dumpcycle.
                                            >> - can't use include/exclude feature of amgtar
                                            >> - I don't know if it is possible to generate an index (-v option)
                                            >> - you can't restore a single file, you must restore the complete stream.
                                            >
                                            > Nick, are you working on an application plugin that uses zfs send/recv
                                            > instead of gtar?
                                            >

                                            Hi Paddy,

                                            Yes I am. I got a zfs send/recv implementation working with 2.6.0 but I
                                            lost it during the upgrade to 2.6.1 (idiot). The follow on application
                                            is nearly ready but currently is partnered with customized version of
                                            amzfs-snapshot to handle the checks & estimates on non-mounted
                                            fileystems. If the my 'amzfs-snapshot-sendrecv; script is not
                                            significantly different to 'amzfs-snapshot' I'll merge them together if
                                            no-one objects.

                                            > thanks,
                                            > Paddy

                                            Regards,

                                            Nick
                                          • Paddy Sreenivasan
                                            ... I think we should keep it separate. amzfs-snapshot will use gnutar to do the actual backup of files from zfs snapshot. This will allow recovery of files.
                                            Message 21 of 23 , Oct 2, 2008
                                            • 0 Attachment
                                              On Thu, Oct 2, 2008 at 5:50 AM, Nick Smith <nick.smith@...> wrote:
                                              >
                                              > Paddy Sreenivasan wrote:
                                              >>
                                              >> On Tue, Aug 19, 2008 at 6:19 AM, Jean-Louis Martineau
                                              >> <martineau@...> wrote:
                                              >>>
                                              >>> Nick Smith wrote:
                                              >>>>
                                              >>>> Hi Jean-Louis,
                                              >>>>
                                              >>>> The archiecture is a lot clearer now - many thanks.
                                              >>>>
                                              >>>> Comments & Questions (in no particular order)
                                              >>>>
                                              >>>> 1. Wouldn't 'zfs send <snapshot> be more efficient than gtar (if
                                              >>>> indexing
                                              >>>> isn't required)? And how about incremental backups - surely ZFS would be
                                              >>>> faster?
                                              >>>
                                              >>> 'zfs send' should be faster but:
                                              >>> - require more space in the pool, snapshot must be kept for the entire
                                              >>> dumpcycle.
                                              >>> - can't use include/exclude feature of amgtar
                                              >>> - I don't know if it is possible to generate an index (-v option)
                                              >>> - you can't restore a single file, you must restore the complete stream.
                                              >>
                                              >> Nick, are you working on an application plugin that uses zfs send/recv
                                              >> instead of gtar?
                                              >>
                                              >
                                              > Hi Paddy,
                                              >
                                              > Yes I am. I got a zfs send/recv implementation working with 2.6.0 but I lost
                                              > it during the upgrade to 2.6.1 (idiot). The follow on application is nearly
                                              > ready but currently is partnered with customized version of amzfs-snapshot
                                              > to handle the checks & estimates on non-mounted fileystems. If the my
                                              > 'amzfs-snapshot-sendrecv; script is not significantly different to
                                              > 'amzfs-snapshot' I'll merge them together if no-one objects.

                                              I think we should keep it separate.

                                              amzfs-snapshot will use gnutar to do the actual backup of files from
                                              zfs snapshot. This will allow recovery of files.

                                              amzfs-snapshot-sendrecv will use zfs send to do the actual backup of
                                              filesystem. This will be similar to a filesystem
                                              dump backup. It can share the estimate and check routines with amzfs-snapshot.

                                              Does amzfs-snapshot-sendrecv use an intermediate file or does it try
                                              to send the zfs send output to the server?

                                              thanks,
                                              Paddy
                                              >
                                              >> thanks,
                                              >> Paddy
                                              >
                                              > Regards,
                                              >
                                              > Nick
                                              >
                                              >



                                              --
                                              Amanda http://amanda.zmanda.com
                                              ZRM for MySQL http://www.zmanda.com/backup-mysql.html
                                            • Jean-Louis Martineau
                                              amzfs-snapshot is a script that run before and after the backup application amgtar. amzfs-snapshot-sendrecv should be an application sending the backup
                                              Message 22 of 23 , Oct 2, 2008
                                              • 0 Attachment
                                                amzfs-snapshot is a script that run before and after the backup
                                                application amgtar.
                                                amzfs-snapshot-sendrecv should be an application sending the backup
                                                directly to the server.

                                                Jean-Louis

                                                Paddy Sreenivasan wrote:
                                                >> Yes I am. I got a zfs send/recv implementation working with 2.6.0 but I lost
                                                >> it during the upgrade to 2.6.1 (idiot). The follow on application is nearly
                                                >> ready but currently is partnered with customized version of amzfs-snapshot
                                                >> to handle the checks & estimates on non-mounted fileystems. If the my
                                                >> 'amzfs-snapshot-sendrecv; script is not significantly different to
                                                >> 'amzfs-snapshot' I'll merge them together if no-one objects.
                                                >>
                                                >
                                                > I think we should keep it separate.
                                                >
                                                > amzfs-snapshot will use gnutar to do the actual backup of files from
                                                > zfs snapshot. This will allow recovery of files.
                                                >
                                                > amzfs-snapshot-sendrecv will use zfs send to do the actual backup of
                                                > filesystem. This will be similar to a filesystem
                                                > dump backup. It can share the estimate and check routines with amzfs-snapshot.
                                                >
                                                > Does amzfs-snapshot-sendrecv use an intermediate file or does it try
                                                > to send the zfs send output to the server?
                                                >
                                                > thanks,
                                                > Paddy
                                                >
                                                >>> thanks,
                                                >>> Paddy
                                                >>>
                                                >> Regards,
                                                >>
                                                >> Nick
                                                >>
                                                >>
                                                >>
                                                >
                                                >
                                                >
                                                >
                                              • Nick Smith
                                                ... This is what I m working on. The issue is that currently that amzfs-snapshot requires that the ZFS filesystem is mounted for the df command to work. So
                                                Message 23 of 23 , Oct 7, 2008
                                                • 0 Attachment
                                                  Paddy Sreenivasan wrote:
                                                  > On Thu, Oct 2, 2008 at 5:50 AM, Nick Smith <nick.smith@...> wrote:
                                                  >> Paddy Sreenivasan wrote:
                                                  >>> On Tue, Aug 19, 2008 at 6:19 AM, Jean-Louis Martineau
                                                  >>> <martineau@...> wrote:
                                                  >>>> Nick Smith wrote:
                                                  >>>>> Hi Jean-Louis,
                                                  >>>>>
                                                  >>>>> The archiecture is a lot clearer now - many thanks.
                                                  >>>>>
                                                  >>>>> Comments & Questions (in no particular order)
                                                  >>>>>
                                                  >>>>> 1. Wouldn't 'zfs send <snapshot> be more efficient than gtar (if
                                                  >>>>> indexing
                                                  >>>>> isn't required)? And how about incremental backups - surely ZFS would be
                                                  >>>>> faster?
                                                  >>>> 'zfs send' should be faster but:
                                                  >>>> - require more space in the pool, snapshot must be kept for the entire
                                                  >>>> dumpcycle.
                                                  >>>> - can't use include/exclude feature of amgtar
                                                  >>>> - I don't know if it is possible to generate an index (-v option)
                                                  >>>> - you can't restore a single file, you must restore the complete stream.
                                                  >>> Nick, are you working on an application plugin that uses zfs send/recv
                                                  >>> instead of gtar?
                                                  >>>
                                                  >> Hi Paddy,
                                                  >>
                                                  >> Yes I am. I got a zfs send/recv implementation working with 2.6.0 but I lost
                                                  >> it during the upgrade to 2.6.1 (idiot). The follow on application is nearly
                                                  >> ready but currently is partnered with customized version of amzfs-snapshot
                                                  >> to handle the checks & estimates on non-mounted fileystems. If the my
                                                  >> 'amzfs-snapshot-sendrecv; script is not significantly different to
                                                  >> 'amzfs-snapshot' I'll merge them together if no-one objects.
                                                  >
                                                  > I think we should keep it separate.
                                                  >
                                                  > amzfs-snapshot will use gnutar to do the actual backup of files from
                                                  > zfs snapshot. This will allow recovery of files.
                                                  >
                                                  > amzfs-snapshot-sendrecv will use zfs send to do the actual backup of
                                                  > filesystem. This will be similar to a filesystem
                                                  > dump backup. It can share the estimate and check routines with amzfs-snapshot.
                                                  >
                                                  This is what I'm working on. The issue is that currently that amzfs-snapshot requires that the ZFS filesystem is mounted for the 'df' command to work. So work is required to support mount and unmounted filesystems that I will do in the next few days (workload permitting).

                                                  > Does amzfs-snapshot-sendrecv use an intermediate file or does it try
                                                  > to send the zfs send output to the server?

                                                  It sends the output directly to the the server. i Got this working briefly with 2.6.0p2 but lost the script :(
                                                  >
                                                  > thanks,
                                                  > Paddy
                                                  >>> thanks,
                                                  >>> Paddy
                                                  >> Regards,
                                                  >>
                                                  >> Nick
                                                  >>
                                                  Regards,

                                                  Nick
                                                Your message has been successfully submitted and would be delivered to recipients shortly.