Loading ...
Sorry, an error occurred while loading the content.
 

transferring unknown number of files at a time.. using Control-M or NDM

Expand Messages
  • Balram
    Hello, We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own
    Message 1 of 8 , Apr 2, 2009
      Hello,

      We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss.txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.

      Thanks,
      Ram
    • Robert Stinnett
      on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative
      Message 2 of 8 , Apr 2, 2009
        on the other end, is the processing beind handled by Control-M?  If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time?  The others will go into a waiting on resource state.  You could even use priorities to control which ones get processed first, second, etc.
         
        Robert


         



         

        To: Control-X@yahoogroups.com
        From: balram.x.nadimpalli@...
        Date: Thu, 2 Apr 2009 20:21:11 +0000
        Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM

        Hello,

        We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss. txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.

        Thanks,
        Ram



      • Balram
        Thanks robert for the response but i think on the other end its not control-M.
        Message 3 of 8 , Apr 2, 2009
          Thanks robert for the response but i think on the other end its not control-M.




          --- In Control-X@yahoogroups.com, Robert Stinnett <robert@...> wrote:
          >
          >
          > on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time? The others will go into a waiting on resource state. You could even use priorities to control which ones get processed first, second, etc.
          >
          >
          >
          > Robert
          >
          >
          >
          >
          >
          >
          >
          >
          >
          > To: Control-X@yahoogroups.com
          > From: balram.x.nadimpalli@...
          > Date: Thu, 2 Apr 2009 20:21:11 +0000
          > Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM
          >
          >
          >
          >
          >
          > Hello,
          >
          > We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss.txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.
          >
          > Thanks,
          > Ram
          >
        • george2ctm
          Would it be more reliable for the processing end to implement logic to handle one file at a time? Lacking that, you can always use a simple shell script /
          Message 4 of 8 , Apr 2, 2009
            Would it be more reliable for the processing end to implement logic to handle one file at a time?

            Lacking that, you can always use a simple shell script / batch script to copy the first file from the directory list.

            For example in Windows
            FOR %i (amex*.txt) DO (copy %i %destination & exit)

            But back to the fundamental problem: how do you know the remote has finished processing and is ready for the next file?


            --- In Control-X@yahoogroups.com, "Balram" <balram.x.nadimpalli@...> wrote:
            >
            > Thanks robert for the response but i think on the other end its not control-M.
            >
            >
            >
            >
            > --- In Control-X@yahoogroups.com, Robert Stinnett <robert@> wrote:
            > >
            > >
            > > on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time? The others will go into a waiting on resource state. You could even use priorities to control which ones get processed first, second, etc.
            > >
            > >
            > >
            > > Robert
            > >
            > >
            > >
            > >
            > >
            > >
            > >
            > >
            > >
            > > To: Control-X@yahoogroups.com
            > > From: balram.x.nadimpalli@
            > > Date: Thu, 2 Apr 2009 20:21:11 +0000
            > > Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM
            > >
            > >
            > >
            > >
            > >
            > > Hello,
            > >
            > > We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss.txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.
            > >
            > > Thanks,
            > > Ram
            > >
            >
          • Balram
            I think on the remote side, the process is designed to handle one file at a time. So as long as there are not more than one file we send to remote end we are
            Message 5 of 8 , Apr 3, 2009
              I think on the remote side, the process is designed to handle one file at a time. So as long as there are not more than one file we send to remote end we are good. So based on conversations.. a gap of about 2-3 would do.

              Thanks







              --- In Control-X@yahoogroups.com, "george2ctm" <george2cfs@...> wrote:
              >
              > Would it be more reliable for the processing end to implement logic to handle one file at a time?
              >
              > Lacking that, you can always use a simple shell script / batch script to copy the first file from the directory list.
              >
              > For example in Windows
              > FOR %i (amex*.txt) DO (copy %i %destination & exit)
              >
              > But back to the fundamental problem: how do you know the remote has finished processing and is ready for the next file?
              >
              >
              > --- In Control-X@yahoogroups.com, "Balram" <balram.x.nadimpalli@> wrote:
              > >
              > > Thanks robert for the response but i think on the other end its not control-M.
              > >
              > >
              > >
              > >
              > > --- In Control-X@yahoogroups.com, Robert Stinnett <robert@> wrote:
              > > >
              > > >
              > > > on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time? The others will go into a waiting on resource state. You could even use priorities to control which ones get processed first, second, etc.
              > > >
              > > >
              > > >
              > > > Robert
              > > >
              > > >
              > > >
              > > >
              > > >
              > > >
              > > >
              > > >
              > > >
              > > > To: Control-X@yahoogroups.com
              > > > From: balram.x.nadimpalli@
              > > > Date: Thu, 2 Apr 2009 20:21:11 +0000
              > > > Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM
              > > >
              > > >
              > > >
              > > >
              > > >
              > > > Hello,
              > > >
              > > > We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss.txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.
              > > >
              > > > Thanks,
              > > > Ram
              > > >
              > >
              >
            • Robert Stinnett
              Can you seperate the AFT job out such that its moving only one of the files and now you have multiple AFT jobs? Then set those jobs up with QR to limit how
              Message 6 of 8 , Apr 3, 2009
                Can you seperate the AFT job out such that its moving only one of the files and now you have multiple AFT jobs?  Then set those jobs up with QR to limit how many are running at a time to one?


                 Robert



                 

                To: Control-X@yahoogroups.com
                From: balram.x.nadimpalli@...
                Date: Fri, 3 Apr 2009 14:29:14 +0000
                Subject: [Control-X] Re: transferring unknown number of files at a time.. using Control-M or NDM


                I think on the remote side, the process is designed to handle one file at a time. So as long as there are not more than one file we send to remote end we are good. So based on conversations. . a gap of about 2-3 would do.

                Thanks

                --- In Control-X@yahoogrou ps.com, "george2ctm" <george2cfs@ ...> wrote:
                >
                > Would it be more reliable for the processing end to implement logic to handle one file at a time?
                >
                > Lacking that, you can always use a simple shell script / batch script to copy the first file from the directory list.
                >
                > For example in Windows
                > FOR %i (amex*.txt) DO (copy %i %destination & exit)
                >
                > But back to the fundamental problem: how do you know the remote has finished processing and is ready for the next file?
                >
                >
                > --- In Control-X@yahoogrou ps.com, "Balram" <balram.x.nadimpall i@> wrote:
                > >
                > > Thanks robert for the response but i think on the other end its not control-M.
                > >
                > >
                > >
                > >
                > > --- In Control-X@yahoogrou ps.com, Robert Stinnett <robert@> wrote:
                > > >
                > > >
                > > > on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time? The others will go into a waiting on resource state. You could even use priorities to control which ones get processed first, second, etc.
                > > >
                > > >
                > > >
                > > > Robert
                > > >
                > > >
                > > >
                > > >
                > > >
                > > >
                > > >
                > > >
                > > >
                > > > To: Control-X@yahoogrou ps.com
                > > > From: balram.x.nadimpalli @
                > > > Date: Thu, 2 Apr 2009 20:21:11 +0000
                > > > Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM
                > > >
                > > >
                > > >
                > > >
                > > >
                > > > Hello,
                > > >
                > > > We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss. txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.
                > > >
                > > > Thanks,
                > > > Ram
                > > >
                > >
                >


              • Balram
                Sounds like a manual process unless i dont understand. Since the number of files are random and we have a file watcher to know that there are files in the
                Message 7 of 8 , Apr 3, 2009
                  Sounds like a manual process unless i dont understand. Since the number of files are random and we have a file watcher to know that there are files in the directory and now how do we create AFT jobs that would transfer these files one by one and to add to the scenario, the file name format would be Amex<yyyymmdd>.txt but a breather here is no two files will have the same date stamp. So, if there are 5 files to be sent then all the 5 files will have different dates. please advise, if that helps any.

                  Thanks






                  --- In Control-X@yahoogroups.com, Robert Stinnett <robert@...> wrote:
                  >
                  >
                  > Can you seperate the AFT job out such that its moving only one of the files and now you have multiple AFT jobs? Then set those jobs up with QR to limit how many are running at a time to one?
                  >
                  >
                  > Robert
                  >
                  >
                  >
                  >
                  >
                  >
                  > To: Control-X@yahoogroups.com
                  > From: balram.x.nadimpalli@...
                  > Date: Fri, 3 Apr 2009 14:29:14 +0000
                  > Subject: [Control-X] Re: transferring unknown number of files at a time.. using Control-M or NDM
                  >
                  >
                  >
                  >
                  >
                  >
                  > I think on the remote side, the process is designed to handle one file at a time. So as long as there are not more than one file we send to remote end we are good. So based on conversations.. a gap of about 2-3 would do.
                  >
                  > Thanks
                  >
                  > --- In Control-X@yahoogroups.com, "george2ctm" <george2cfs@> wrote:
                  > >
                  > > Would it be more reliable for the processing end to implement logic to handle one file at a time?
                  > >
                  > > Lacking that, you can always use a simple shell script / batch script to copy the first file from the directory list.
                  > >
                  > > For example in Windows
                  > > FOR %i (amex*.txt) DO (copy %i %destination & exit)
                  > >
                  > > But back to the fundamental problem: how do you know the remote has finished processing and is ready for the next file?
                  > >
                  > >
                  > > --- In Control-X@yahoogroups.com, "Balram" <balram.x.nadimpalli@> wrote:
                  > > >
                  > > > Thanks robert for the response but i think on the other end its not control-M.
                  > > >
                  > > >
                  > > >
                  > > >
                  > > > --- In Control-X@yahoogroups.com, Robert Stinnett <robert@> wrote:
                  > > > >
                  > > > >
                  > > > > on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time? The others will go into a waiting on resource state. You could even use priorities to control which ones get processed first, second, etc.
                  > > > >
                  > > > >
                  > > > >
                  > > > > Robert
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > > To: Control-X@yahoogroups.com
                  > > > > From: balram.x.nadimpalli@
                  > > > > Date: Thu, 2 Apr 2009 20:21:11 +0000
                  > > > > Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > >
                  > > > > Hello,
                  > > > >
                  > > > > We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss.txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.
                  > > > >
                  > > > > Thanks,
                  > > > > Ram
                  > > > >
                  > > >
                  > >
                  >
                • george2ctm
                  It still goes back to the same question: when do you send the next file? So let s assume the trigger to send is handled by operator. Then, JobFW is one-time
                  Message 8 of 8 , Apr 3, 2009
                    It still goes back to the same question: when do you send the next file? So let's assume the trigger to send is handled by operator. Then,

                    JobFW is one-time filewatcher job to trigger JobMOVE. JobFW watches for files in directory INCOMING.

                    JobMOVE is a cyclic job to trigger JobAFT. It moves one file at a time using shell command from directory INCOMING to directory TRANSFER. Shell command can move file according FIFO or in whatever sequence.

                    JobAFT is a cyclic AFT job to FTP file from directory TRANSFER to remote. Upon complete, AFT moves transferred file away from the directory.

                    Each cyclic jobs deletes its own IN condition.

                    The first transfer is automatically triggered by JobFW upon discovering file. Subsequent transfer will require operator to add condition to JobMOVE.

                    Again, if the processing engine does not have mechanism to protect itself from multiple files, there is no way to prevent operator from sending the next file prematurely.

                    I hope this helps.


                    --- In Control-X@yahoogroups.com, "Balram" <balram.x.nadimpalli@...> wrote:
                    >
                    >
                    > Sounds like a manual process unless i dont understand. Since the number of files are random and we have a file watcher to know that there are files in the directory and now how do we create AFT jobs that would transfer these files one by one and to add to the scenario, the file name format would be Amex<yyyymmdd>.txt but a breather here is no two files will have the same date stamp. So, if there are 5 files to be sent then all the 5 files will have different dates. please advise, if that helps any.
                    >
                    > Thanks
                    >
                    >
                    >
                    >
                    >
                    >
                    > --- In Control-X@yahoogroups.com, Robert Stinnett <robert@> wrote:
                    > >
                    > >
                    > > Can you seperate the AFT job out such that its moving only one of the files and now you have multiple AFT jobs? Then set those jobs up with QR to limit how many are running at a time to one?
                    > >
                    > >
                    > > Robert
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > > To: Control-X@yahoogroups.com
                    > > From: balram.x.nadimpalli@
                    > > Date: Fri, 3 Apr 2009 14:29:14 +0000
                    > > Subject: [Control-X] Re: transferring unknown number of files at a time.. using Control-M or NDM
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > > I think on the remote side, the process is designed to handle one file at a time. So as long as there are not more than one file we send to remote end we are good. So based on conversations.. a gap of about 2-3 would do.
                    > >
                    > > Thanks
                    > >
                    > > --- In Control-X@yahoogroups.com, "george2ctm" <george2cfs@> wrote:
                    > > >
                    > > > Would it be more reliable for the processing end to implement logic to handle one file at a time?
                    > > >
                    > > > Lacking that, you can always use a simple shell script / batch script to copy the first file from the directory list.
                    > > >
                    > > > For example in Windows
                    > > > FOR %i (amex*.txt) DO (copy %i %destination & exit)
                    > > >
                    > > > But back to the fundamental problem: how do you know the remote has finished processing and is ready for the next file?
                    > > >
                    > > >
                    > > > --- In Control-X@yahoogroups.com, "Balram" <balram.x.nadimpalli@> wrote:
                    > > > >
                    > > > > Thanks robert for the response but i think on the other end its not control-M.
                    > > > >
                    > > > >
                    > > > >
                    > > > >
                    > > > > --- In Control-X@yahoogroups.com, Robert Stinnett <robert@> wrote:
                    > > > > >
                    > > > > >
                    > > > > > on the other end, is the processing beind handled by Control-M? If so, since you said there is a process for handling each file, why not setup a quantitative resource so that only one processing job flow can be running at a time? The others will go into a waiting on resource state. You could even use priorities to control which ones get processed first, second, etc.
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > > Robert
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > > To: Control-X@yahoogroups.com
                    > > > > > From: balram.x.nadimpalli@
                    > > > > > Date: Thu, 2 Apr 2009 20:21:11 +0000
                    > > > > > Subject: [Control-X] transferring unknown number of files at a time.. using Control-M or NDM
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > >
                    > > > > > Hello,
                    > > > > >
                    > > > > > We have a scenario where we get unknown number of files at a time or one file in a 15 minute cycle of batch processing. Now, each file will have its own date time stamp embed in a unique file name format ( eg: amexyyyymmddhhmmss.txt ) and needs to be shipped over to another location where there is a process setup to process each of these files. The question is how can we copy/move them so we send each file separately. we can't do a wild card copy using AFT because on the other end only one file needs to get processed at a time otherwise it gets mixed up or something like that. Can anyone have any suggestions as to how we can accomplish this.
                    > > > > >
                    > > > > > Thanks,
                    > > > > > Ram
                    > > > > >
                    > > > >
                    > > >
                    > >
                    >
                  Your message has been successfully submitted and would be delivered to recipients shortly.