Loading ...
Sorry, an error occurred while loading the content.

Re: [NH] Estimating Storage Usage

Expand Messages
  • Larry Hamilton
    Ray, It might be that they have automatic web server logs saved until your delete them from the server. Usually there is a log directory on the same level as
    Message 1 of 8 , Nov 16, 2005
    • 0 Attachment
      Ray,

      It might be that they have automatic web server logs saved until your
      delete them from the server.

      Usually there is a log directory on the same level as the htdocs
      directory (location of web pages).

      In my experience, there is either one log file per day, or one per month.

      If you do not clear these out regularly, they can add up significantly.

      You may want to back them up to CD in case you ever want to evaluate
      them. They are good for helping you to improve the navigation of your site.

      HTH,

      Larry Hamilton
      Kairos Computer Solutions
      http://www.kairoscomputers.com/
      Sales Affiliate for Grisoft Anti-Virus

      R Shapp wrote:

      >HI Group,
      >
      >Can anyone suggest a tool for estimating storage used for each subfolder
      >within a website?
      >
      >My hosting service says our site is near the maximum allowed in our service
      >plan (300MB). I believe that number to be way incorrect because I mirror most
      >of our site on my local PC. Windows Explorer shows a little more than 2300
      >files in 98 folders consuming 63MB in local storage.
      >
      >I'm looking for a tool that will examine our site online and report the
      >storage used by the 100 or so folders and sub-folders in our site.
      >
      >If this isn't the proper group for this kind of question, I'd appreciate your
      >suggestion as to a more appropriate venue.
      >
      >Thanks for the help.
      >
      >Ray Shapp
      >
    • R Shapp
      Hi Larry, Our Hosting service did once keep such logs for us, but they appear to have stopped
      Message 2 of 8 , Nov 17, 2005
      • 0 Attachment
        Hi Larry,

        <<It might be that they have automatic web server logs saved>>

        Our Hosting service did once keep such logs for us, but they appear to have
        stopped doing that. I manually looked at the files in all the top level
        folders (about 25 folders), and I didn't see any log files. The company is
        HostCentric. I am very surprised to find that Customer Service says all they
        can tell me is our total usage.

        I'm still looking for some kind of tool for analyzing my storage usage.

        Thanks for your comments.

        Ray Shapp
      • acummingsus
        ... all they ... http://www.google.com/search?q=HostCentric&start=0&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official From that, I see
        Message 3 of 8 , Nov 17, 2005
        • 0 Attachment
          --- In ntb-html@yahoogroups.com, R Shapp <ras45@o...> wrote:
          > The company is
          > HostCentric. I am very surprised to find that Customer Service says
          all they
          > can tell me is our total usage.
          >
          > I'm still looking for some kind of tool for analyzing my storage usage.

          http://www.google.com/search?q=HostCentric&start=0&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official

          From that, I see Win or Linux server available. I saw "control panel".

          What sort of access do you have? The control panel may liken to
          Webmin? (webmin works with Linux but I'm unaware if it can work on
          Win) If so, in Linux server likely can do a user's permissions (what's
          allowed) as far as command line.

          ls -la

          df

          (the df gives disk file useage, might not limit itself to your
          account/allocation.

          Those commands usually will not work on a Windows box.

          Do you have shell access? ftp access? ssh?

          With ftp, you could view all the files (just to verify that something
          didn't accidentally duplicate a bunch of files (or hopefully not
          cracked into for to container of someone else's files too)

          Alan.
        • Don - htmlfixit.com
          ... If it were me, I would download everything off of the server and then run analysis of it. There are some good tools for that. Look at what is old and
          Message 4 of 8 , Nov 17, 2005
          • 0 Attachment
            R Shapp wrote:
            > Hi Larry,
            >
            > <<It might be that they have automatic web server logs saved>>
            >
            > Our Hosting service did once keep such logs for us, but they appear to have
            > stopped doing that. I manually looked at the files in all the top level
            > folders (about 25 folders), and I didn't see any log files. The company is
            > HostCentric. I am very surprised to find that Customer Service says all they
            > can tell me is our total usage.
            >
            > I'm still looking for some kind of tool for analyzing my storage usage.
            >
            > Thanks for your comments.
            >
            > Ray Shapp
            >
            >
            If it were me, I would download everything off of the server and then
            run analysis of it. There are some good tools for that. Look at what
            is old and what is getting no hits on your server logs.

            I use xenu link sleuth (available on net ... I think it is care ware).
            It is a great tool. It finds broken links and the like.

            It will run your server and tell you about dead things that are not
            linked to anything. Getting rid of some of those things might help.

            We found that we had a program generating images as part of our spam
            fighting program for comments that weren't getting deleted. We needed
            to add a cron job (actually I decided to wipe them over a certain age
            using the same php script that was creating them instead of a cron job
            in the end) to get them cleaned out regularly.

            Logs and error logs can get significant in size over time.
          • thefrank
            ditto regarding logfile remarks. a good FTP or SSH client will provide an easy way to report the storage of each file and folder. you may need to calculate
            Message 5 of 8 , Nov 17, 2005
            • 0 Attachment
              ditto regarding logfile remarks.

              a good FTP or SSH client will provide an easy way to report the
              storage of each file and folder. you may need to calculate these.

              what I would add is about email. on our servers email storage is
              included in the total amount used. if I set a 2GB limit for each
              account, one domain may reach that limit with only 10MB of webpage
              storage but another domain would approach the limit with 500MB.

              this is because in the first case they are consuming their total
              allotment with email. they may want 10 email accounts with unlimited
              use. the emailboxes may be stuffed full of thousands of 100-day-old
              messages left on the mailserver because they prefer to use web-based
              email [IMAP] rather than a local POP client. some of these configs
              require to manually delete email from the server whereas a POP client
              usually empties the mailserver each time the mail is downloaded.

              many users attach 10 family photos each one being 2MB because their
              new digital camera takes such swell pics at 2400DPI. well there is
              20MB consumed. now send the same email to a list of 10 family
              members, with the same attachments. now there is 200MB consumed,
              although only briefly while the mail is in the queue waiting to be
              sent, but hopefully you get the idea about email and storage. this is
              why many of the free email servers limit storage to 5 or 10MB, and
              many times your email gets kicked.

              so look for 2 things first:

              1. logfiles, as mentioned by others...

              2. email accounts and useage.

              your admin widget may allow you to manipulate these.

              simple solutions would include:

              1. [unix] set a crontab to zip all logfiles daily and config
              webalizer.conf [or other] to report from zipped archive. just this
              should reduce logfiles to 10% of uncompressed filesize.

              2. rotate old logfiles off the server. monthly or even weekly.

              3. monitor email useage and educate offender users to post photo
              albums to a webpage and just email a link to the page instead of all
              those attachments that send the photo album to many recipients.

              4. limit the mailbox storage of repeat offenders.

              regards,

              tf
            • loro
              ... Do you have their CP? It says Monitor bandwidth and disk space . Otherwise get something like this
              Message 6 of 8 , Nov 17, 2005
              • 0 Attachment
                R Shapp wrote:
                >I'm still looking for some kind of tool for analyzing my storage usage.

                Do you have their CP? It says "Monitor bandwidth and disk space".
                <http://www.hostcentric.com/popups/control_panel.bml>

                Otherwise get something like this (I have no experience with this script.
                Just the first I found.)
                <http://www.perlservices.net/en/programs/site_file_lister/>

                Lotta
              • R Shapp
                Hi Lotta, thefrank, Don, Alan, Thank you for all the great suggestions. I ll definitely try them, but for now, the problem is solved quite conventionally. Our
                Message 7 of 8 , Nov 17, 2005
                • 0 Attachment
                  Hi Lotta, thefrank, Don, Alan,

                  Thank you for all the great suggestions. I'll definitely try them, but for
                  now, the problem is solved quite conventionally.

                  Our site was being attacked by one or more spam robots. We had been getting
                  annoyance spams added to our guestbook, so last Spring, I installed one of the
                  more sophisticated guestbook handlers. It worked for a while, but it, too,
                  was getting two or three inappropriate emails each week. As a substitute, I
                  created a Yahoo discussion group like this one for all our club members.
                  After it became accepted, I deleted the menu navigation on our website to the
                  guestbook page. I didn't, however, disable the guestbook application.

                  Yesterday, in a manual search of our site using an FTP application, I was
                  amazed to discover that the messages folder of our guestbook contained more
                  than 33,000 spam emails. Examining the dates of the spam messages, some days
                  had no entries, but since Sunday I was receiving thousands of spams each day,
                  including 10,200 spams on November 16th.

                  I purged the spam, and deleted the guestbook software.. Even though our
                  account has a storage limit of 300MB, HostCentric had allowed it to
                  temporarily balloon to 478MB. After the purge (and the deletion of some old
                  traffic statistics files as suggested here), we now are back to 68MB in use.
                  Crisis averted!

                  The tools and techniques you mentioned are good to know about. Thank you for
                  all your help.

                  Ray Shapp
                Your message has been successfully submitted and would be delivered to recipients shortly.