Loading ...
Sorry, an error occurred while loading the content.
 

Re: [PBML] moving files to existing structure

Expand Messages
  • Dave Gray
    ... rsync will do what you need (run the following from /somedir/traces): rsync -vur . /data/traces The command-line switches are v=verbose, u=update,
    Message 1 of 15 , Aug 3, 2004
      On Tue, 3 Aug 2004 14:25:53 -0500 (CDT), Paul Archer <tigger@...> wrote:
      > I've got a bunch (thousands) of files in a structure like this:
      >
      > /somedir/traces/2004/08/03/abc/longfilename.abi
      >
      > and I need to move them to another directory structure that already has some
      > directories in place:
      >
      > /data/traces/2004/08/03/abc/longfilename.abi
      >
      > where /data/traces/2004/08/03 (for example) may or may not exist.

      rsync will do what you need (run the following from /somedir/traces):
      rsync -vur . /data/traces

      The command-line switches are v=verbose, u=update, r=recursive, which
      should be all you need. The manpage has more info if you want to do
      some fancier stuff.

      There's probably a perl interface to it (I can't access CPAN at the
      moment for some reason), but that seems like overkill for this
      specific task.

      Cheers,
      Dave
    • Paul Archer
      ... Maybe you misunderstood me. ... I ve got a bunch (thousands) of files in a structure like this: /somedir/traces/2004/08/03/abc/longfilename.abi and I need
      Message 2 of 15 , Aug 3, 2004
        12:37pm, merlyn@... wrote:

        > >>>>> "Paul" == Paul Archer <tigger@...> writes:
        >
        > Paul> So the question is: is there a module/script whatever out there
        > Paul> that would help with this? I'd rather not have to write a
        > Paul> solution from scratch if I can avoid it. (It would have to find
        > Paul> each filename, check to see if the directory structure existed
        > Paul> already, create it if doesn't, and then move the file.)
        >
        > It's about eight lines of code. If you don't want to write it
        > from scratch, maybe you can hire someone to write eight lines
        > of code for you.
        >
        > perldoc File::Basename (especially basename and dirname)
        > perldoc -f mkdir
        > perldoc File::Copy
        >
        > Truly, you are demonstrating laziness beyond reproach.
        >

        Maybe you misunderstood me.

        (Here's the bit you chopped off my original post):
        ----include----
        I've got a bunch (thousands) of files in a structure like this:

        /somedir/traces/2004/08/03/abc/longfilename.abi

        and I need to move them to another directory structure that already has some
        directories in place:

        /data/traces/2004/08/03/abc/longfilename.abi

        where /data/traces/2004/08/03 (for example) may or may not exist.


        The problem comes in when trying to simply 'mv' the files, as I can't say
        mv /somedir/traces /data/traces
        because the 'traces' directory already exists.
        ---end include----


        I can't simply do a mv (either a Unix mv or a Perl mv/move) because of the
        existing structure. I have to preserve the structure from 'traces' on. I
        can't simply dump the files into the same directory.
        In other words:

        mkdir -p /tmp/temp_dir/traces/2004/08/03/
        mkdir -p /tmp/final_dir/traces/2004/08/03
        touch /tmp/temp_dir/traces/2004/08/03/file_to_be_moved.abi

        # doesn't work
        mv /tmp/temp_dir/traces /tmp/final_dir

        # works, but requires the directory structure to be checked/created
        # for every file moved
        mv /tmp/temp_dir/traces/2004/08/03/file_to_be_moved.abi \
        /tmp/final_dir/traces/2004/08/03

        Checking/creating the directory structure wouldn't be so bad, but I've got
        to move over 3.6 million files.

        Now, it may be that there's a simple solution. It may be that I'm being lazy
        ("beyond reproach", even) for asking for help rather than slogging it out
        completely on my own (but I don't think so). But I doubt a workable solution
        is going to fit in 8 lines of code.


        Paul Archer
      • Jeff Eggen
        ... How about 4 lines? cd /somedir/traces tar cf /tmp/somefile.tar * cd /data/traces tar xf /tmp/somefile.tar This will overwrite stuff with the same name, but
        Message 3 of 15 , Aug 3, 2004
          >>> tigger@... 03/08/2004 3:08:26 pm >>>
          >Now, it may be that there's a simple solution. It may be that I'm being lazy
          >("beyond reproach", even) for asking for help rather than slogging it out
          >completely on my own (but I don't think so). But I doubt a workable solution
          >is going to fit in 8 lines of code.

          How about 4 lines?

          cd /somedir/traces
          tar cf /tmp/somefile.tar *
          cd /data/traces
          tar xf /tmp/somefile.tar

          This will overwrite stuff with the same name, but won't hurt anything that already exists in /data/traces but doesn't in /somedir/traces.

          Another solution would be a one-liner with find & cpio, but I can't remember it off the top of my head. I think it's something like

          find /somedir/traces -print | cpio -dumpV /data/traces

          Another way would be to copy stuff with the cp command, and then remove the old stuff.

          Another way would be rsync.

          Another way would be, if you don't care about the existing stuff under /data/traces with name conflicts, then you can just do a

          mv /somedir/traces/* /data/traces/

          If none of these does what you want, then I must be misunderstanding the problem, in which case I apologize. All of these solutions will cause problems with name conflicts, but that isn't hard to overcome if you have a naming convention for archiving pre-existing files.

          Hope one of these helps,

          Jeff Eggen
          IT Programmer Analyst
          Saskatchewan Government Insurance
          Ph (306) 751-1795
          email jeggen@...

          ************DISCLAIMER*************
          This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee, please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that using, disclosing, copying or distributing the contents of this information is strictly prohibited.
          ************DISCLAIMER*************
        • Paul Archer
          ... Thanks for the suggestion. It s close, but unfortunately, I can t copy the files and then delete the originals (I don t have the space--the files are about
          Message 4 of 15 , Aug 3, 2004
            3:54pm, Dave Gray wrote:

            > On Tue, 3 Aug 2004 14:25:53 -0500 (CDT), Paul Archer <tigger@...> wrote:
            > > I've got a bunch (thousands) of files in a structure like this:
            > >
            > > /somedir/traces/2004/08/03/abc/longfilename.abi
            > >
            > > and I need to move them to another directory structure that already has some
            > > directories in place:
            > >
            > > /data/traces/2004/08/03/abc/longfilename.abi
            > >
            > > where /data/traces/2004/08/03 (for example) may or may not exist.
            >
            > rsync will do what you need (run the following from /somedir/traces):
            > rsync -vur . /data/traces
            >
            > The command-line switches are v=verbose, u=update, r=recursive, which
            > should be all you need. The manpage has more info if you want to do
            > some fancier stuff.
            >
            > There's probably a perl interface to it (I can't access CPAN at the
            > moment for some reason), but that seems like overkill for this
            > specific task.
            >

            Thanks for the suggestion. It's close, but unfortunately, I can't copy the
            files and then delete the originals (I don't have the space--the files are
            about 3/4 of a terabyte total). And I can't see any way to get rsync to
            delete a file after it has sync'ed it to the new structure (not surprising,
            there).

            Thanks anyway,

            Paul
          • Paul Archer
            ... As I mentioned in another post, my problem is that due to the sheer volume of the data 3/4 of a terabyte or so, I simply can t do anything that copies then
            Message 5 of 15 , Aug 3, 2004
              3:23pm, Jeff Eggen wrote:

              > >>> tigger@... 03/08/2004 3:08:26 pm >>>
              > >Now, it may be that there's a simple solution. It may be that I'm being lazy
              > >("beyond reproach", even) for asking for help rather than slogging it out
              > >completely on my own (but I don't think so). But I doubt a workable solution
              > >is going to fit in 8 lines of code.
              >
              > How about 4 lines?
              >
              > cd /somedir/traces
              > tar cf /tmp/somefile.tar *
              > cd /data/traces
              > tar xf /tmp/somefile.tar
              >
              > This will overwrite stuff with the same name, but won't hurt anything that already exists in /data/traces but doesn't in /somedir/traces.
              >
              > Another solution would be a one-liner with find & cpio, but I can't remember it off the top of my head. I think it's something like
              >
              > find /somedir/traces -print | cpio -dumpV /data/traces
              >
              > Another way would be to copy stuff with the cp command, and then remove the old stuff.
              >
              > Another way would be rsync.
              >
              > Another way would be, if you don't care about the existing stuff under /data/traces with name conflicts, then you can just do a
              >
              > mv /somedir/traces/* /data/traces/
              >
              > If none of these does what you want, then I must be misunderstanding the problem, in which case I apologize. All of these solutions will cause problems with name conflicts, but that isn't hard to overcome if you have a naming convention for archiving pre-existing files.
              >
              As I mentioned in another post, my problem is that due to the sheer volume
              of the data 3/4 of a terabyte or so, I simply can't do anything that copies
              then removes. I just don't have the room.
              Good suggestions otherwise, though.


              Paul Archer
            • merlyn@stonehenge.com
              ... Paul Now, it may be that there s a simple solution. It may be that Paul I m being lazy ( beyond reproach , even) for asking for help Paul rather than
              Message 6 of 15 , Aug 3, 2004
                >>>>> "Paul" == Paul Archer <tigger@...> writes:

                Paul> Now, it may be that there's a simple solution. It may be that
                Paul> I'm being lazy ("beyond reproach", even) for asking for help
                Paul> rather than slogging it out completely on my own (but I don't
                Paul> think so). But I doubt a workable solution is going to fit in 8
                Paul> lines of code.

                Here's the exact code that would do the job. The first two initialize
                your parameters. I don't count those, because those could have been
                envars or @ARGV. Three of the lines are "use", using two core modules
                and one CPAN module. The remaining 5 lines perform precisely the task
                you want, but I've "X'ed" those out because you haven't sent me a
                check yet.

                my $SRC = "/somedir/traces";
                my $DST = "/data/traces";
                XXX XXXXXXXXXXXXX
                XXX XXXXXXXXXXX
                XXX XXXXXXXXXXXXXXX
                XXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX X
                XXX XXXX X XXX XX XXXXXXXXXXXXXXXX
                XXXXXXXXXXXXXXXXXXXXXX
                XXXXXXXXXXXXXXXX
                X

                Eight lines of code exactly. Yeah, that was a guess when I first
                though of it, but the code bears it out.

                Now stop your whining. Either learn some Perl, or hire someone
                to turn those X's back into code.

                --
                Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
                <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
                Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
                See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
              • Jeff 'japhy' Pinyan
                ... Couldn t you do: use File::Basename fileparse ; use File::Path mkpath ; for (@list_of_files) { my ($file, $path) = fileparse($_); (my $new_path = $path)
                Message 7 of 15 , Aug 3, 2004
                  On Aug 3, Paul Archer said:

                  >I've got a bunch (thousands) of files in a structure like this:
                  >
                  >/somedir/traces/2004/08/03/abc/longfilename.abi
                  >
                  >and I need to move them to another directory structure that already has some
                  >directories in place:
                  >
                  >/data/traces/2004/08/03/abc/longfilename.abi
                  >
                  >where /data/traces/2004/08/03 (for example) may or may not exist.

                  Couldn't you do:

                  use File::Basename 'fileparse';
                  use File::Path 'mkpath';

                  for (@list_of_files) {
                  my ($file, $path) = fileparse($_);
                  (my $new_path = $path) =~ s{^/somedir}{/data};
                  mkpath $new_path
                  or warn "couldn't mkpath $new_path: $!";
                  rename $_ => "$new_path/$file"
                  or warn "couldn't mv $_ to $new_path/$file: $!";
                  }

                  --
                  Jeff "japhy" Pinyan % How can we ever be the sold short or
                  RPI Acacia Brother #734 % the cheated, we who for every service
                  http://japhy.perlmonk.org/ % have long ago been overpaid?
                  http://www.perlmonks.org/ % -- Meister Eckhart
                • Jeff Eggen
                  ... Then the move suggestion should work for you. They didn t all involve copying. cd /somedir/traces for i in `find . -print | sort` do ([ -d $i ] && [ -x
                  Message 8 of 15 , Aug 3, 2004
                    >>> tigger@... 03/08/2004 3:27:27 pm >>>
                    >As I mentioned in another post, my problem is that due to the sheer volume
                    >of the data 3/4 of a terabyte or so, I simply can't do anything that copies
                    >then removes. I just don't have the room.
                    >Good suggestions otherwise, though.

                    Then the move suggestion should work for you. They didn't all involve copying.

                    cd /somedir/traces
                    for i in `find . -print | sort`
                    do
                    ([ -d $i ] && [ -x /data/traces/$i ]) || mkdir /data/traces/$i
                    [ -f /data/traces/$i ] && mv /data/traces/$i /data/traces/${i}.archived
                    [ -f $i ] && mv $i /data/traces/$i
                    done

                    Kind of verbose at 7 lines, but it'll get the job done, and won't require any more space. Explicit for those who want explicitness, I guess. The second line that begins with "test" should take care of file name conflicts, but won't do anything about directories. I tried it here on one of my rigs, and it seems to work.

                    And for the record: you never mentioned the space requirements being a restriction. You just mentioned that there were a lot of files, which doesn't mean much.

                    Hope this helps,

                    Jeff Eggen
                    IT Programmer Analyst
                    Saskatchewan Government Insurance
                    Ph (306) 751-1795
                    email jeggen@...

                    ************DISCLAIMER*************
                    This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee, please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that using, disclosing, copying or distributing the contents of this information is strictly prohibited.
                    ************DISCLAIMER*************
                  • Paul Archer
                    ... I ve seen you slap other people down before. I ve even been guilty of following suit. Glad it s happened to me now, so I know what it feels like, and I can
                    Message 9 of 15 , Aug 4, 2004
                      Yesterday, Randal L. Schwartz wrote:

                      > >>>>> "Paul" == Paul Archer <tigger@...> writes:
                      >
                      > Paul> Now, it may be that there's a simple solution. It may be that
                      > Paul> I'm being lazy ("beyond reproach", even) for asking for help
                      > Paul> rather than slogging it out completely on my own (but I don't
                      > Paul> think so). But I doubt a workable solution is going to fit in 8
                      > Paul> lines of code.
                      >
                      > Here's the exact code that would do the job. The first two initialize
                      > your parameters. I don't count those, because those could have been
                      > envars or @ARGV. Three of the lines are "use", using two core modules
                      > and one CPAN module. The remaining 5 lines perform precisely the task
                      > you want, but I've "X'ed" those out because you haven't sent me a
                      > check yet.
                      >
                      > my $SRC = "/somedir/traces";
                      > my $DST = "/data/traces";
                      > XXX XXXXXXXXXXXXX
                      > XXX XXXXXXXXXXX
                      > XXX XXXXXXXXXXXXXXX
                      > XXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX X
                      > XXX XXXX X XXX XX XXXXXXXXXXXXXXXX
                      > XXXXXXXXXXXXXXXXXXXXXX
                      > XXXXXXXXXXXXXXXX
                      > X
                      >
                      > Eight lines of code exactly. Yeah, that was a guess when I first
                      > though of it, but the code bears it out.
                      >
                      > Now stop your whining. Either learn some Perl, or hire someone
                      > to turn those X's back into code.
                      >


                      I've seen you slap other people down before. I've even been guilty of
                      following suit. Glad it's happened to me now, so I know what it feels like,
                      and I can make sure I don't play the asshole again the next time I feel like
                      stomping on someone for what I feel is an unreasonable post.

                      FWIW, I did word my original post poorly, and left out some important
                      details--like the fact that I was looking for hints, not 8 lines of
                      carefully crafted code by the Master; or that I had to run this rename on
                      over 3.6 million files comprising almost a terabyte of data and 4 days of
                      tape restores, and I was really nervous about doing something wrong, as well
                      as worried about efficiency.

                      Perhaps had I mentioned those facts, you would have been less harsh. But
                      it's a bit late now for both of us.

                      Paul Archer
                    • Paul Archer
                      I had no idea the mkpath subroutine was out there. I ended up doing a system ( mkdir , -p , $dir ) but next time I ll definitely use mkpath instead. Thanks
                      Message 10 of 15 , Aug 4, 2004
                        I had no idea the mkpath subroutine was out there. I ended up doing a
                        system ("mkdir", "-p", "$dir")
                        but next time I'll definitely use mkpath instead.
                        Thanks for the tip!

                        Paul

                        Yesterday, Jeff 'japhy' Pinyan wrote:

                        > On Aug 3, Paul Archer said:
                        >
                        > >I've got a bunch (thousands) of files in a structure like this:
                        > >
                        > >/somedir/traces/2004/08/03/abc/longfilename.abi
                        > >
                        > >and I need to move them to another directory structure that already has some
                        > >directories in place:
                        > >
                        > >/data/traces/2004/08/03/abc/longfilename.abi
                        > >
                        > >where /data/traces/2004/08/03 (for example) may or may not exist.
                        >
                        > Couldn't you do:
                        >
                        > use File::Basename 'fileparse';
                        > use File::Path 'mkpath';
                        >
                        > for (@list_of_files) {
                        > my ($file, $path) = fileparse($_);
                        > (my $new_path = $path) =~ s{^/somedir}{/data};
                        > mkpath $new_path
                        > or warn "couldn't mkpath $new_path: $!";
                        > rename $_ => "$new_path/$file"
                        > or warn "couldn't mv $_ to $new_path/$file: $!";
                        > }
                        >
                        > --
                        > Jeff "japhy" Pinyan % How can we ever be the sold short or
                        > RPI Acacia Brother #734 % the cheated, we who for every service
                        > http://japhy.perlmonk.org/ % have long ago been overpaid?
                        > http://www.perlmonks.org/ % -- Meister Eckhart
                        >
                        >
                        >
                        >
                        > Unsubscribing info is here: http://help.yahoo.com/help/us/groups/groups-32.html
                        > Yahoo! Groups Links
                        >
                        >
                        >
                        >
                        >

                        --------------------------------------------------------------
                        "I'd cry/cry for the future/but I wouldn't get anything done."
                        ----------------------Hunters + Collectors--------------------
                      • Paul Archer
                        ... Thanks for the code. I ended up doing it in Perl, but it s always nice to see how different people handle the same task. ... You are quite right. I should
                        Message 11 of 15 , Aug 4, 2004
                          > >>> tigger@... 03/08/2004 3:27:27 pm >>>
                          > >As I mentioned in another post, my problem is that due to the sheer volume
                          > >of the data 3/4 of a terabyte or so, I simply can't do anything that copies
                          > >then removes. I just don't have the room.
                          > >Good suggestions otherwise, though.
                          >
                          > Then the move suggestion should work for you. They didn't all involve copying.
                          >
                          > cd /somedir/traces
                          > for i in `find . -print | sort`
                          > do
                          > ([ -d $i ] && [ -x /data/traces/$i ]) || mkdir /data/traces/$i
                          > [ -f /data/traces/$i ] && mv /data/traces/$i /data/traces/${i}.archived
                          > [ -f $i ] && mv $i /data/traces/$i
                          > done
                          >
                          > Kind of verbose at 7 lines, but it'll get the job done, and won't require
                          > any more space. Explicit for those who want explicitness, I guess. The
                          > second line that begins with "test" should take care of file name
                          > conflicts, but won't do anything about directories. I tried it here on
                          > one of my rigs, and it seems to work.
                          >
                          Thanks for the code. I ended up doing it in Perl, but it's always nice to
                          see how different people handle the same task.



                          > And for the record: you never mentioned the space requirements being a
                          > restriction. You just mentioned that there were a lot of files, which
                          > doesn't mean much.
                          >

                          You are quite right. I should have mentioned it. The whole reason for my
                          original post, after all, was that I had millions of files to move, and was
                          afraid that it would take forever (and I was pretty nervous about messing
                          something up, as that could mean having to pull this data off of tape
                          *again*, a process that took 4 days the first time).

                          Paul Archer
                        • merlyn@stonehenge.com
                          ... Paul FWIW, I did word my original post poorly, and left out some important Paul details--like the fact that I was looking for hints, And I gave you those
                          Message 12 of 15 , Aug 4, 2004
                            >>>>> "Paul" == Paul Archer <tigger@...> writes:

                            Paul> FWIW, I did word my original post poorly, and left out some important
                            Paul> details--like the fact that I was looking for hints,

                            And I gave you those hints. My first response was a series
                            of perldoc commands that would have led you to most of the solution
                            I posted.

                            Paul> not 8 lines of
                            Paul> carefully crafted code by the Master;

                            And while your solution might not have been 8 lines, that was part of
                            my first response to show you that the task is NOT HARD.

                            Paul> or that I had to run this rename on
                            Paul> over 3.6 million files comprising almost a terabyte of data and 4 days of
                            Paul> tape restores, and I was really nervous about doing something wrong, as well
                            Paul> as worried about efficiency.

                            And if you weren't confident to do that yourself, you should have
                            HIRED SOMEONE. Not asked for FREE LABOR on a technical discussion
                            list.

                            Paul> Perhaps had I mentioned those facts, you would have been less harsh. But
                            Paul> it's a bit late now for both of us.

                            I'm only harsh when I see people misunderstanding the purpose of a
                            technical mailing list, or wanting to be SO LAZY to get others to do
                            their work for them.

                            I have absolutely no problem with people showing effort, then getting
                            stuck, then asking for the next step here. I help people like that
                            here (and all over) all the time. Maybe you should try to understand
                            why you didn't come across like that, but rather like someone who
                            wanted the world for free, instead of working toward your goal.

                            --
                            Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
                            <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
                            Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
                            See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
                          • Jeff Eggen
                            ... Yeah, TMTOWTDI and all that. I usually don t use Perl for file moves & copies, unless they are part of a larger whole. Besides, a lot of times seeing a
                            Message 13 of 15 , Aug 9, 2004
                              >>> tigger@... 04/08/2004 8:21:36 am >>>
                              >Thanks for the code. I ended up doing it in Perl, but it's always nice to
                              >see how different people handle the same task.

                              Yeah, TMTOWTDI and all that. I usually don't use Perl for file moves & copies, unless they are part of a larger whole. Besides, a lot of times seeing a solution in the shell will help me visualise a solution in Perl, and vice-versa.

                              Have fun!

                              Jeff Eggen
                              IT Programmer Analyst
                              Saskatchewan Government Insurance
                              Ph (306) 751-1795
                              email jeggen@...

                              ************DISCLAIMER*************
                              This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the named addressee, please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that using, disclosing, copying or distributing the contents of this information is strictly prohibited.
                              ************DISCLAIMER*************
                            Your message has been successfully submitted and would be delivered to recipients shortly.