Loading ...
Sorry, an error occurred while loading the content.

Moving RRDtool files to internal memory - where?

Expand Messages
  • d0nv
    I m putting together a vacation home monitoring system for my in-laws based on an uNSLUng 6.8 slug. I have it now up and running producing graphs from the
    Message 1 of 10 , Jan 4, 2011
    View Source
    • 0 Attachment
      I'm putting together a vacation home monitoring system for my in-laws based on an uNSLUng 6.8 slug. I have it now up and running producing graphs from the 1-wire sensors via temploggerd/RRDtool and displaying them via a separate instance of thttpd, so I can open just this one port to internet access. It is currently writing data to the USB memory stick for this purpose and I know this is going to wear it out quickly.

      What I'd like to do is have it write to internal memory for web display and then periodically back the files up to the USB stick for persistence and also recovery of the data upon reboot.

      I've scoured the nslu2-linux.org site but can't seem to find a definitive answer on what amount of internal space is available and what file system path to use to access it for this purpose *on uNSLUng 6.8*. Is this info available?

      Any/all tips appreciated. I'm coming up against deadline on this project (they leave shortly for several months, this is intended to cover them during that period).

      Believed applicable specifics on my system below. If these listings are both in the same units, I'm concerned there won't be enough space available...

      Thanks.

      # df -k
      Filesystem 1k-blocks Used Available Use% Mounted on
      rootfs 285603 179417 103240 63% /
      /dev/sda1 6528 6344 184 97% /initrd
      /dev/sda1 285603 179417 103240 63% /
      /dev/sda1 285603 179417 103240 63% /share/flash/data
      /dev/sda2 74843 4157 69913 6% /share/flash/conf

      Files and sizes that would need to be stored on internal memory for website display:

      # ls -l
      -rw-r--r-- 1 website everyone 17327 Jan 4 21:52 280097A7020000tempe.png
      -rw-r--r-- 1 website everyone 19061 Jan 5 00:25 2875BAA7020000tempe.png
      -rw-r--r-- 1 website everyone 19206 Jan 5 00:25 28848BD5020000tempe.png
      -rw-r--r-- 1 website everyone 19712 Jan 5 00:24 all_temperature.png
      -rw-rw-rw- 1 root root 366912 Jan 5 00:25 temperature.rrd
    • Emil Granström
      Not quite the answer you are looking for but still: From what I have read current USB sticks will last several years even if they are written to quite often. I
      Message 2 of 10 , Jan 5, 2011
      View Source
      • 0 Attachment
        Not quite the answer you are looking for but still:
        From what I have read current USB sticks will last several years even if they are written to quite often.
        I have been using my slug with OWFS and RRDTool data on USB stick for about 9 months without any problems so far.

        Brg
         Emil
        On 2011-01-05 06:56, d0nv wrote:
         

        I'm putting together a vacation home monitoring system for my in-laws based on an uNSLUng 6.8 slug. I have it now up and running producing graphs from the 1-wire sensors via temploggerd/RRDtool and displaying them via a separate instance of thttpd, so I can open just this one port to internet access. It is currently writing data to the USB memory stick for this purpose and I know this is going to wear it out quickly.

        What I'd like to do is have it write to internal memory for web display and then periodically back the files up to the USB stick for persistence and also recovery of the data upon reboot.

        I've scoured the nslu2-linux.org site but can't seem to find a definitive answer on what amount of internal space is available and what file system path to use to access it for this purpose *on uNSLUng 6.8*. Is this info available?

        Any/all tips appreciated. I'm coming up against deadline on this project (they leave shortly for several months, this is intended to cover them during that period).

        Believed applicable specifics on my system below. If these listings are both in the same units, I'm concerned there won't be enough space available...

        Thanks.

        # df -k
        Filesystem 1k-blocks Used Available Use% Mounted on
        rootfs 285603 179417 103240 63% /
        /dev/sda1 6528 6344 184 97% /initrd
        /dev/sda1 285603 179417 103240 63% /
        /dev/sda1 285603 179417 103240 63% /share/flash/data
        /dev/sda2 74843 4157 69913 6% /share/flash/conf

        Files and sizes that would need to be stored on internal memory for website display:

        # ls -l
        -rw-r--r-- 1 website everyone 17327 Jan 4 21:52 280097A7020000tempe.png
        -rw-r--r-- 1 website everyone 19061 Jan 5 00:25 2875BAA7020000tempe.png
        -rw-r--r-- 1 website everyone 19206 Jan 5 00:25 28848BD5020000tempe.png
        -rw-r--r-- 1 website everyone 19712 Jan 5 00:24 all_temperature.png
        -rw-rw-rw- 1 root root 366912 Jan 5 00:25 temperature.rrd

      • Mike Westerhof
        ... Not really. And they re cheap. I ll also mention that modern, good-quality USB flash devices do a good job of wear-leveling, so you can effectively
        Message 3 of 10 , Jan 5, 2011
        View Source
        • 0 Attachment
          d0nv wrote:
          > I'm putting together a vacation home monitoring system for my in-laws based on an uNSLUng 6.8 slug. I have it now up and running producing graphs from the 1-wire sensors via temploggerd/RRDtool and displaying them via a separate instance of thttpd, so I can open just this one port to internet access. It is currently writing data to the USB memory stick for this purpose and I know this is going to wear it out quickly.
          >
          Not really. And they're cheap. I'll also mention that modern,
          good-quality USB flash devices do a good job of wear-leveling, so you
          can effectively increase the lifetime of the device if you only use a
          small percentage of the device's space. So, pick a 4GB unit, for
          example, for 12 USD at your local Walmart store, and expect it to last
          for years.
          > What I'd like to do is have it write to internal memory for web display and then periodically back the files up to the USB stick for persistence and also recovery of the data upon reboot.
          >
          > I've scoured the nslu2-linux.org site but can't seem to find a definitive answer on what amount of internal space is available and what file system path to use to access it for this purpose *on uNSLUng 6.8*. Is this info available?
          >
          Nope. Unfortunately, you picked the wrong firmware for that purpose --
          SlugOS would have given you the in-memory filesystem you want; it's not
          something that works out-of-the-box with Unslung. (I haven't checked,
          so I can't even say that you can find the stuff you need in the feeds to
          make it work at all with Unslung, frankly.)
          > Any/all tips appreciated. I'm coming up against deadline on this project (they leave shortly for several months, this is intended to cover them during that period).
          >
          > Believed applicable specifics on my system below. If these listings are both in the same units, I'm concerned there won't be enough space available...
          >
          > Thanks.
          >
          > # df -k
          > Filesystem 1k-blocks Used Available Use% Mounted on
          > rootfs 285603 179417 103240 63% /
          > /dev/sda1 6528 6344 184 97% /initrd
          > /dev/sda1 285603 179417 103240 63% /
          > /dev/sda1 285603 179417 103240 63% /share/flash/data
          > /dev/sda2 74843 4157 69913 6% /share/flash/conf
          >
          > Files and sizes that would need to be stored on internal memory for website display:
          >
          > # ls -l
          > -rw-r--r-- 1 website everyone 17327 Jan 4 21:52 280097A7020000tempe.png
          > -rw-r--r-- 1 website everyone 19061 Jan 5 00:25 2875BAA7020000tempe.png
          > -rw-r--r-- 1 website everyone 19206 Jan 5 00:25 28848BD5020000tempe.png
          > -rw-r--r-- 1 website everyone 19712 Jan 5 00:24 all_temperature.png
          > -rw-rw-rw- 1 root root 366912 Jan 5 00:25 temperature.rrd
          >
          >
          Small files. Put them on a modern USB flash device, and don't worry
          about it.

          -Mike (mwester)
        • d0nv
          Thanks. I guess I could always just prepare them a new flash drive on a predictable schedule (eg: once a year) if they would last that long. There s more than
          Message 4 of 10 , Jan 5, 2011
          View Source
          • 0 Attachment
            Thanks. I guess I could always just prepare them a new flash drive on a predictable schedule (eg: once a year) if they would last that long. There's more than enough space on the current 512MB(!) stick.

            Still interested in the original internal storage info, should anyone have it.

            I guess another alternative is to have the slug sftp/rsync the rrdfile data to my shared hosting server periodically, solely for backup. Does anyone have a good recipe for this?

            --- In nslu2-linux@yahoogroups.com, Emil Granstr�m <emilg@...> wrote:
            >
            > Not quite the answer you are looking for but still:
            > From what I have read current USB sticks will last several years even
            > if they are written to quite often.
            > I have been using my slug with OWFS and RRDTool data on USB stick for
            > about 9 months without any problems so far.
            >
            > Brg
            > Emil
          • d0nv
            Thanks Mike. Picking unslung was simply the path of least resistance, I m repurposing a machine that was already unslung and had everything else set up to my
            Message 5 of 10 , Jan 5, 2011
            View Source
            • 0 Attachment
              Thanks Mike. Picking unslung was simply the path of least resistance, I'm repurposing a machine that was already unslung and had everything else set up to my needs as a result.

              I'm happy to go with the cheap drives/replace periodically approach vs. spending more time on this. Would still love a ssh/rsync remote backup recipe pointer for unslung, if one exists.

              Thanks again.

              --- In nslu2-linux@yahoogroups.com, Mike Westerhof <mwester@...> wrote:
              >
              > d0nv wrote:
              > > I'm putting together a vacation home monitoring system for my in-laws based on an uNSLUng 6.8 slug. I have it now up and running producing graphs from the 1-wire sensors via temploggerd/RRDtool and displaying them via a separate instance of thttpd, so I can open just this one port to internet access. It is currently writing data to the USB memory stick for this purpose and I know this is going to wear it out quickly.
              > >
              > Not really. And they're cheap. I'll also mention that modern,
              > good-quality USB flash devices do a good job of wear-leveling, so you
              > can effectively increase the lifetime of the device if you only use a
              > small percentage of the device's space. So, pick a 4GB unit, for
              > example, for 12 USD at your local Walmart store, and expect it to last
              > for years.
              > > What I'd like to do is have it write to internal memory for web display and then periodically back the files up to the USB stick for persistence and also recovery of the data upon reboot.
              > >
              > > I've scoured the nslu2-linux.org site but can't seem to find a definitive answer on what amount of internal space is available and what file system path to use to access it for this purpose *on uNSLUng 6.8*. Is this info available?
              > >
              > Nope. Unfortunately, you picked the wrong firmware for that purpose --
              > SlugOS would have given you the in-memory filesystem you want; it's not
              > something that works out-of-the-box with Unslung. (I haven't checked,
              > so I can't even say that you can find the stuff you need in the feeds to
              > make it work at all with Unslung, frankly.)
              > > Any/all tips appreciated. I'm coming up against deadline on this project (they leave shortly for several months, this is intended to cover them during that period).
              > >
              > > Believed applicable specifics on my system below. If these listings are both in the same units, I'm concerned there won't be enough space available...
              > >
              > > Thanks.
              > >
              > > # df -k
              > > Filesystem 1k-blocks Used Available Use% Mounted on
              > > rootfs 285603 179417 103240 63% /
              > > /dev/sda1 6528 6344 184 97% /initrd
              > > /dev/sda1 285603 179417 103240 63% /
              > > /dev/sda1 285603 179417 103240 63% /share/flash/data
              > > /dev/sda2 74843 4157 69913 6% /share/flash/conf
              > >
              > > Files and sizes that would need to be stored on internal memory for website display:
              > >
              > > # ls -l
              > > -rw-r--r-- 1 website everyone 17327 Jan 4 21:52 280097A7020000tempe.png
              > > -rw-r--r-- 1 website everyone 19061 Jan 5 00:25 2875BAA7020000tempe.png
              > > -rw-r--r-- 1 website everyone 19206 Jan 5 00:25 28848BD5020000tempe.png
              > > -rw-r--r-- 1 website everyone 19712 Jan 5 00:24 all_temperature.png
              > > -rw-rw-rw- 1 root root 366912 Jan 5 00:25 temperature.rrd
              > >
              > >
              > Small files. Put them on a modern USB flash device, and don't worry
              > about it.
              >
              > -Mike (mwester)
              >
            • Doug
              Using 1-wire with Debian on NSLU2 here. I do a udp broadcast every 15 minutes to the monitoring site and also have it setup to accept http access to
              Message 6 of 10 , Jan 5, 2011
              View Source
              • 0 Attachment
                Using 1-wire with Debian on NSLU2 here. I do a udp broadcast every 15 minutes to the monitoring site and also have it setup to accept http access to temperatures but I have not implemented rdtool. I would be interested in your configuration there if you would be willing to share it.

                Doug
                 
                Doug Crompton
                WA3DSP
                www.crompton.com



                From: d0nv <nslu2-yahoo@...>
                To: nslu2-linux@yahoogroups.com
                Sent: Wed, January 5, 2011 12:56:46 AM
                Subject: [nslu2-linux] Moving RRDtool files to internal memory - where?

                 

                I'm putting together a vacation home monitoring system for my in-laws based on an uNSLUng 6.8 slug. I have it now up and running producing graphs from the 1-wire sensors via temploggerd/RRDtool and displaying them via a separate instance of thttpd, so I can open just this one port to internet access. It is currently writing data to the USB memory stick for this purpose and I know this is going to wear it out quickly.

                What I'd like to do is have it write to internal memory for web display and then periodically back the files up to the USB stick for persistence and also recovery of the data upon reboot.

                I've scoured the nslu2-linux.org site but can't seem to find a definitive answer on what amount of internal space is available and what file system path to use to access it for this purpose *on uNSLUng 6.8*. Is this info available?

                Any/all tips appreciated. I'm coming up against deadline on this project (they leave shortly for several months, this is intended to cover them during that period).

                Believed applicable specifics on my system below. If these listings are both in the same units, I'm concerned there won't be enough space available...

                Thanks.

                # df -k
                Filesystem 1k-blocks Used Available Use% Mounted on
                rootfs 285603 179417 103240 63% /
                /dev/sda1 6528 6344 184 97% /initrd
                /dev/sda1 285603 179417 103240 63% /
                /dev/sda1 285603 179417 103240 63% /share/flash/data
                /dev/sda2 74843 4157 69913 6% /share/flash/conf

                Files and sizes that would need to be stored on internal memory for website display:

                # ls -l
                -rw-r--r-- 1 website everyone 17327 Jan 4 21:52 280097A7020000tempe.png
                -rw-r--r-- 1 website everyone 19061 Jan 5 00:25 2875BAA7020000tempe.png
                -rw-r--r-- 1 website everyone 19206 Jan 5 00:25 28848BD5020000tempe.png
                -rw-r--r-- 1 website everyone 19712 Jan 5 00:24 all_temperature.png
                -rw-rw-rw- 1 root root 366912 Jan 5 00:25 temperature.rrd

              • d0nv
                Thanks for mentioning the UDP broadcast. For me, the two machines will be on separate networks across the internet so I don t think that will work in my
                Message 7 of 10 , Jan 5, 2011
                View Source
                • 0 Attachment
                  Thanks for mentioning the UDP broadcast. For me, the two machines will be on separate networks across the internet so I don't think that will work in my application.

                  We're running different firmware, so our rrdtool set ups will likely be different... that said, the key is installing temploggerd, which will drag along and set up rrdtool with the associated files and page displays/graphs.

                  On unslung, this was as simple as:
                  ipkg update
                  ipkg install temploggerd

                  The default configuration on unslung pretty much worked out of the box for LAN access of the 1-wire data pages/graphs. OTOH, owfs config files required tweaking to work with my iButtonLink LinkUSB adaptors, but it appears you have that sorted.

                  I did extensive tweaking of the temploggerd config files afterwards for my purpose. Because I'm exposing this to internet access, I wanted to use an obscure port for port forwarding to minimize random port scans load. That necessitated I move the files elsewhere, etc.

                  I have a blog post I'll be updating shortly to detail my overall solution. See http://veino.com/blog/?p=518 if interested.

                  --- In nslu2-linux@yahoogroups.com, Doug <dsc3507@...> wrote:
                  >
                  > Using 1-wire with Debian on NSLU2 here. I do a udp broadcast every 15 minutes to
                  > the monitoring site and also have it setup to accept http access to temperatures
                  > but I have not implemented rdtool. I would be interested in your configuration
                  > there if you would be willing to share it.
                  >
                  > Doug
                  >
                  > Doug Crompton
                  > WA3DSP
                  > www.crompton.com
                  >
                • Doug
                  Thanks for the rundown. I had considered using the tol but wanted a quick and easy way to get the temps from one place to another. I wrote a udpserver in Perl
                  Message 8 of 10 , Jan 5, 2011
                  View Source
                  • 0 Attachment
                    Thanks for the rundown. I had considered using the tol but wanted a quick and easy way to get the temps from one place to another.

                    I wrote a udpserver in Perl that runs and just listens for data on a port at my home location. This location has a static IP so the place I want to monitor sends a udp packet every 15 minutes with the latest 1wire data. There is no handshaking.  This is over the internet and it works fine. I have not missed a 15 minute update and even if I did miss one it would be a trivial problem. On the home router I just open up the selected port to the machine running the server. Security has not been a problem. I only except data with a certain PW in it, I only accept 256 characters or less, and I only allow one reception every minute. Since there is no handshake response anyone that happens to send data to that udp port would have no knowledge they are even talking to an active machine.  Simple but it works.

                    Doug
                     
                    Doug Crompton
                    WA3DSP
                    www.crompton.com



                    From: d0nv <nslu2-yahoo@...>
                    To: nslu2-linux@yahoogroups.com
                    Sent: Wed, January 5, 2011 1:28:53 PM
                    Subject: [nslu2-linux] Re: Moving RRDtool files to internal memory - where?

                     



                    Thanks for mentioning the UDP broadcast. For me, the two machines will be on separate networks across the internet so I don't think that will work in my application.

                    We're running different firmware, so our rrdtool set ups will likely be different... that said, the key is installing temploggerd, which will drag along and set up rrdtool with the associated files and page displays/graphs.

                    On unslung, this was as simple as:
                    ipkg update
                    ipkg install temploggerd

                    The default configuration on unslung pretty much worked out of the box for LAN access of the 1-wire data pages/graphs. OTOH, owfs config files required tweaking to work with my iButtonLink LinkUSB adaptors, but it appears you have that sorted.

                    I did extensive tweaking of the temploggerd config files afterwards for my purpose. Because I'm exposing this to internet access, I wanted to use an obscure port for port forwarding to minimize random port scans load. That necessitated I move the files elsewhere, etc.

                    I have a blog post I'll be updating shortly to detail my overall solution. See http://veino.com/blog/?p=518 if interested.

                    --- In nslu2-linux@yahoogroups.com, Doug <dsc3507@...> wrote:
                    >
                    > Using 1-wire with Debian on NSLU2 here. I do a udp broadcast every 15 minutes to
                    > the monitoring site and also have it setup to accept http access to temperatures
                    > but I have not implemented rdtool. I would be interested in your configuration
                    > there if you would be willing to share it.
                    >
                    > Doug
                    >
                    > Doug Crompton
                    > WA3DSP
                    > www.crompton.com
                    >

                  • clerew5
                    ... As a matter of interest, will it spread stuff out even into neighbouring partitions. I have a 2GB USBstick (twice the size I really need, but you can t buy
                    Message 9 of 10 , Jan 6, 2011
                    View Source
                    • 0 Attachment
                      --- In nslu2-linux@yahoogroups.com, Mike Westerhof <mwester@...> wrote:

                      > Not really. And they're cheap. I'll also mention that modern,
                      > good-quality USB flash devices do a good job of wear-leveling, so you
                      > can effectively increase the lifetime of the device if you only use a
                      > small percentage of the device's space. So, pick a 4GB unit, for
                      > example, for 12 USD at your local Walmart store, and expect it to last
                      > for years.

                      As a matter of interest, will it spread stuff out even into neighbouring partitions. I have a 2GB USBstick (twice the size I really need, but you can't buy them smaller anymore). It is partitioned into 4 lots of 250MB and one of 1GB. Only two of the partitions see anything like heavy use.

                      Will that usage spread out into the areas nominally reserved for the empty 1GB? I.e. are the addresses where stuff is actually stored entirely different from their nominal addresses on the "disk", with some lookup table in between?
                    • Mike Westerhof
                      ... The wear-leveling is done at the block-level, and it is unaware of things like partitions. So it is fair to say that if the read-write cycle count remains
                      Message 10 of 10 , Jan 6, 2011
                      View Source
                      • 0 Attachment
                        clerew5 wrote:
                        > --- In nslu2-linux@yahoogroups.com, Mike Westerhof <mwester@...> wrote:
                        > Not really. And they're cheap. I'll also mention that modern,
                        > good-quality USB flash devices do a good job of wear-leveling, so you
                        > can effectively increase the lifetime of the device if you only use a
                        > small percentage of the device's space. So, pick a 4GB unit, for
                        > example, for 12 USD at your local Walmart store, and expect it to last
                        > for years.
                        >
                        > As a matter of interest, will it spread stuff out even into neighbouring partitions. I have a 2GB USBstick (twice the size I really need, but you can't buy them smaller anymore). It is partitioned into 4 lots of 250MB and one of 1GB. Only two of the partitions see anything like heavy use.
                        >
                        > Will that usage spread out into the areas nominally reserved for the empty 1GB? I.e. are the addresses where stuff is actually stored entirely different from their nominal addresses on the "disk", with some lookup table in between?
                        >

                        The wear-leveling is done at the block-level, and it is unaware of
                        things like partitions. So it is fair to say that if the read-write
                        cycle count remains the same, a 4GB stick will last twice as long as a
                        2GB stick, regardless of how the disk is partitioned.

                        -Mike
                      Your message has been successfully submitted and would be delivered to recipients shortly.