Loading ...
Sorry, an error occurred while loading the content.
 

Estimating Storage Usage

Expand Messages
  • R Shapp
    HI Group, Can anyone suggest a tool for estimating storage used for each subfolder within a website? My hosting service says our site is near the maximum
    Message 1 of 8 , Nov 16, 2005
      HI Group,

      Can anyone suggest a tool for estimating storage used for each subfolder
      within a website?

      My hosting service says our site is near the maximum allowed in our service
      plan (300MB). I believe that number to be way incorrect because I mirror most
      of our site on my local PC. Windows Explorer shows a little more than 2300
      files in 98 folders consuming 63MB in local storage.

      I'm looking for a tool that will examine our site online and report the
      storage used by the 100 or so folders and sub-folders in our site.

      If this isn't the proper group for this kind of question, I'd appreciate your
      suggestion as to a more appropriate venue.

      Thanks for the help.

      Ray Shapp
    • Larry Hamilton
      Ray, It might be that they have automatic web server logs saved until your delete them from the server. Usually there is a log directory on the same level as
      Message 2 of 8 , Nov 16, 2005
        Ray,

        It might be that they have automatic web server logs saved until your
        delete them from the server.

        Usually there is a log directory on the same level as the htdocs
        directory (location of web pages).

        In my experience, there is either one log file per day, or one per month.

        If you do not clear these out regularly, they can add up significantly.

        You may want to back them up to CD in case you ever want to evaluate
        them. They are good for helping you to improve the navigation of your site.

        HTH,

        Larry Hamilton
        Kairos Computer Solutions
        http://www.kairoscomputers.com/
        Sales Affiliate for Grisoft Anti-Virus

        R Shapp wrote:

        >HI Group,
        >
        >Can anyone suggest a tool for estimating storage used for each subfolder
        >within a website?
        >
        >My hosting service says our site is near the maximum allowed in our service
        >plan (300MB). I believe that number to be way incorrect because I mirror most
        >of our site on my local PC. Windows Explorer shows a little more than 2300
        >files in 98 folders consuming 63MB in local storage.
        >
        >I'm looking for a tool that will examine our site online and report the
        >storage used by the 100 or so folders and sub-folders in our site.
        >
        >If this isn't the proper group for this kind of question, I'd appreciate your
        >suggestion as to a more appropriate venue.
        >
        >Thanks for the help.
        >
        >Ray Shapp
        >
      • R Shapp
        Hi Larry, Our Hosting service did once keep such logs for us, but they appear to have stopped
        Message 3 of 8 , Nov 17, 2005
          Hi Larry,

          <<It might be that they have automatic web server logs saved>>

          Our Hosting service did once keep such logs for us, but they appear to have
          stopped doing that. I manually looked at the files in all the top level
          folders (about 25 folders), and I didn't see any log files. The company is
          HostCentric. I am very surprised to find that Customer Service says all they
          can tell me is our total usage.

          I'm still looking for some kind of tool for analyzing my storage usage.

          Thanks for your comments.

          Ray Shapp
        • acummingsus
          ... all they ... http://www.google.com/search?q=HostCentric&start=0&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official From that, I see
          Message 4 of 8 , Nov 17, 2005
            --- In ntb-html@yahoogroups.com, R Shapp <ras45@o...> wrote:
            > The company is
            > HostCentric. I am very surprised to find that Customer Service says
            all they
            > can tell me is our total usage.
            >
            > I'm still looking for some kind of tool for analyzing my storage usage.

            http://www.google.com/search?q=HostCentric&start=0&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official

            From that, I see Win or Linux server available. I saw "control panel".

            What sort of access do you have? The control panel may liken to
            Webmin? (webmin works with Linux but I'm unaware if it can work on
            Win) If so, in Linux server likely can do a user's permissions (what's
            allowed) as far as command line.

            ls -la

            df

            (the df gives disk file useage, might not limit itself to your
            account/allocation.

            Those commands usually will not work on a Windows box.

            Do you have shell access? ftp access? ssh?

            With ftp, you could view all the files (just to verify that something
            didn't accidentally duplicate a bunch of files (or hopefully not
            cracked into for to container of someone else's files too)

            Alan.
          • Don - htmlfixit.com
            ... If it were me, I would download everything off of the server and then run analysis of it. There are some good tools for that. Look at what is old and
            Message 5 of 8 , Nov 17, 2005
              R Shapp wrote:
              > Hi Larry,
              >
              > <<It might be that they have automatic web server logs saved>>
              >
              > Our Hosting service did once keep such logs for us, but they appear to have
              > stopped doing that. I manually looked at the files in all the top level
              > folders (about 25 folders), and I didn't see any log files. The company is
              > HostCentric. I am very surprised to find that Customer Service says all they
              > can tell me is our total usage.
              >
              > I'm still looking for some kind of tool for analyzing my storage usage.
              >
              > Thanks for your comments.
              >
              > Ray Shapp
              >
              >
              If it were me, I would download everything off of the server and then
              run analysis of it. There are some good tools for that. Look at what
              is old and what is getting no hits on your server logs.

              I use xenu link sleuth (available on net ... I think it is care ware).
              It is a great tool. It finds broken links and the like.

              It will run your server and tell you about dead things that are not
              linked to anything. Getting rid of some of those things might help.

              We found that we had a program generating images as part of our spam
              fighting program for comments that weren't getting deleted. We needed
              to add a cron job (actually I decided to wipe them over a certain age
              using the same php script that was creating them instead of a cron job
              in the end) to get them cleaned out regularly.

              Logs and error logs can get significant in size over time.
            • thefrank
              ditto regarding logfile remarks. a good FTP or SSH client will provide an easy way to report the storage of each file and folder. you may need to calculate
              Message 6 of 8 , Nov 17, 2005
                ditto regarding logfile remarks.

                a good FTP or SSH client will provide an easy way to report the
                storage of each file and folder. you may need to calculate these.

                what I would add is about email. on our servers email storage is
                included in the total amount used. if I set a 2GB limit for each
                account, one domain may reach that limit with only 10MB of webpage
                storage but another domain would approach the limit with 500MB.

                this is because in the first case they are consuming their total
                allotment with email. they may want 10 email accounts with unlimited
                use. the emailboxes may be stuffed full of thousands of 100-day-old
                messages left on the mailserver because they prefer to use web-based
                email [IMAP] rather than a local POP client. some of these configs
                require to manually delete email from the server whereas a POP client
                usually empties the mailserver each time the mail is downloaded.

                many users attach 10 family photos each one being 2MB because their
                new digital camera takes such swell pics at 2400DPI. well there is
                20MB consumed. now send the same email to a list of 10 family
                members, with the same attachments. now there is 200MB consumed,
                although only briefly while the mail is in the queue waiting to be
                sent, but hopefully you get the idea about email and storage. this is
                why many of the free email servers limit storage to 5 or 10MB, and
                many times your email gets kicked.

                so look for 2 things first:

                1. logfiles, as mentioned by others...

                2. email accounts and useage.

                your admin widget may allow you to manipulate these.

                simple solutions would include:

                1. [unix] set a crontab to zip all logfiles daily and config
                webalizer.conf [or other] to report from zipped archive. just this
                should reduce logfiles to 10% of uncompressed filesize.

                2. rotate old logfiles off the server. monthly or even weekly.

                3. monitor email useage and educate offender users to post photo
                albums to a webpage and just email a link to the page instead of all
                those attachments that send the photo album to many recipients.

                4. limit the mailbox storage of repeat offenders.

                regards,

                tf
              • loro
                ... Do you have their CP? It says Monitor bandwidth and disk space . Otherwise get something like this
                Message 7 of 8 , Nov 17, 2005
                  R Shapp wrote:
                  >I'm still looking for some kind of tool for analyzing my storage usage.

                  Do you have their CP? It says "Monitor bandwidth and disk space".
                  <http://www.hostcentric.com/popups/control_panel.bml>

                  Otherwise get something like this (I have no experience with this script.
                  Just the first I found.)
                  <http://www.perlservices.net/en/programs/site_file_lister/>

                  Lotta
                • R Shapp
                  Hi Lotta, thefrank, Don, Alan, Thank you for all the great suggestions. I ll definitely try them, but for now, the problem is solved quite conventionally. Our
                  Message 8 of 8 , Nov 17, 2005
                    Hi Lotta, thefrank, Don, Alan,

                    Thank you for all the great suggestions. I'll definitely try them, but for
                    now, the problem is solved quite conventionally.

                    Our site was being attacked by one or more spam robots. We had been getting
                    annoyance spams added to our guestbook, so last Spring, I installed one of the
                    more sophisticated guestbook handlers. It worked for a while, but it, too,
                    was getting two or three inappropriate emails each week. As a substitute, I
                    created a Yahoo discussion group like this one for all our club members.
                    After it became accepted, I deleted the menu navigation on our website to the
                    guestbook page. I didn't, however, disable the guestbook application.

                    Yesterday, in a manual search of our site using an FTP application, I was
                    amazed to discover that the messages folder of our guestbook contained more
                    than 33,000 spam emails. Examining the dates of the spam messages, some days
                    had no entries, but since Sunday I was receiving thousands of spams each day,
                    including 10,200 spams on November 16th.

                    I purged the spam, and deleted the guestbook software.. Even though our
                    account has a storage limit of 300MB, HostCentric had allowed it to
                    temporarily balloon to 478MB. After the purge (and the deletion of some old
                    traffic statistics files as suggested here), we now are back to 68MB in use.
                    Crisis averted!

                    The tools and techniques you mentioned are good to know about. Thank you for
                    all your help.

                    Ray Shapp
                  Your message has been successfully submitted and would be delivered to recipients shortly.