Loading ...
Sorry, an error occurred while loading the content.
 

RE: [PBML] Re: Eval and memory problems

Expand Messages
  • Boyle, Christopher
    perldoc -quote ________________________________ From: perl-beginner@yahoogroups.com [mailto:perl-beginner@yahoogroups.com] On Behalf Of computerdribble Sent:
    Message 1 of 15 , Jan 8, 2008
      perldoc -quote



      ________________________________

      From: perl-beginner@yahoogroups.com
      [mailto:perl-beginner@yahoogroups.com] On Behalf Of computerdribble
      Sent: Tuesday, January 08, 2008 3:38 PM
      To: perl-beginner@yahoogroups.com
      Subject: [PBML] Re: Eval and memory problems




      Thank you.

      I went onto the faq.perl.org site and searched through all of that
      and nothing came back.

      This is one of the problems and reasons I have stayed away from perl
      for so long. There's SO much to it and it HAS its OWN quirkiness that
      outpaces other languages.

      but again, thank you!

      Now, one other question....

      this works:
      $text =~ s/(\$\w+)/$1/eeg

      it takes;
      $aa="hello";
      $bb="goodbye";

      and inserts it into;

      $text = "( ( $aa =~ /lo/ ) and ($bb =~ /ye/ ) )";

      perfectly to create:

      ( ( hello =~ /lo/ ) and ( goodbye =~ /ye/ ) )

      notice the problem ? No quotes around the $aa expanded variable. So the
      if ( @{ [ $text ] } )

      always fails.

      I'm NOT a regex person and stumble my way through it so far.. but this
      is beyond me at the moment. (yes more reading is required)

      How do I get the expansion (or can it be done? ) to automatically put
      the quotes around the variables expanded?

      I can add the quotes into the database around the variables, but then
      its a cludge.

      thanks again and thanks in advance...

      joe

      --- In perl-beginner@yahoogroups.com
      <mailto:perl-beginner%40yahoogroups.com> , merlyn@... wrote:
      >
      > >>>>> "computerdribble" == computerdribble <computerdribble@...>
      writes:
      >
      > computerdribble> Nope , not in there. there is nothing in the FAQ
      concerning Macro
      > computerdribble> expansion.
      >
      > By the way, it's on *YOUR DISK*. No need to go off to the net for
      the FAQ.
      > Type "perldoc perlfaq". And you can search it too, which I will now
      do for
      > you, to show you the FAQ *does* cover what you're looking for.
      >
      > I suggest you read the subject lines of "perldoc perlfaq" and
      "perldoc perl"
      > weekly until you can recite them from memory. Perl comes with a
      HUGE amount
      > of info, but it's pointless if you don't know what's there.
      >
      > $ perldoc -q expand
      >
      > Found in /usr/libdata/perl5/pod/perlfaq4.pod
      > How do I expand function calls in a string?
      > (contributed by brian d foy)
      >
      > This is documented in perlref, and although it's not the easiest
      thing
      > to read, it does work. In each of these examples, we call the
      function
      > inside the braces used to dereference a reference. If we have a more
      > than one return value, we can construct and dereference an anonymous
      > array. In this case, we call the function in list context.
      >
      > print "The time values are @{ [localtime] }.\n";
      >
      > If we want to call the function in scalar context, we have to do
      a bit
      > more work. We can really have any code we like inside the
      braces, so we
      > simply have to end with the scalar reference, although how you
      do that
      > is up to you, and you can use code inside the braces.
      >
      > print "The time is ${\(scalar localtime)}.\n"
      >
      > print "The time is ${ my $x = localtime; \$x }.\n";
      >
      > If your function already returns a reference, you don't need to
      create
      > the reference yourself.
      >
      > sub timestamp { my $t = localtime; \$t }
      >
      > print "The time is ${ timestamp() }.\n";
      >
      > The "Interpolation" module can also do a lot of magic for you.
      You can
      > specify a variable name, in this case "E", to set up a tied hash
      that
      > does the interpolation for you. It has several other methods to
      do this
      > as well.
      >
      > use Interpolation E => 'eval';
      > print "The time values are $E{localtime()}.\n";
      >
      > In most cases, it is probably easier to simply use string
      concatenation,
      > which also forces scalar context.
      >
      > print "The time is " . localtime . ".\n";
      >
      > How do I expand tabs in a string?
      > You can do it yourself:
      >
      > 1 while $string =~ s/\t+/' ' x (length($&) * 8 - length($`)
      % 8)/e;
      >
      > Or you can just use the Text::Tabs module (part of the standard Perl
      > distribution).
      >
      > use Text::Tabs;
      > @expanded_lines = expand(@lines_with_tabs);
      >
      > How can I expand variables in text strings?
      > Let's assume that you have a string that contains placeholder
      variables.
      >
      > $text = 'this has a $foo in it and a $bar';
      >
      > You can use a substitution with a double evaluation. The first
      /e turns
      > $1 into $foo, and the second /e turns $foo into its value. You
      may want
      > to wrap this in an "eval": if you try to get the value of an
      undeclared
      > variable while running under "use strict", you get a fatal error.
      >
      > eval { $text =~ s/(\$\w+)/$1/eeg };
      > die if $@;
      >
      > It's probably better in the general case to treat those variables as
      > entries in some special hash. For example:
      >
      > %user_defs = (
      > foo => 23,
      > bar => 19,
      > );
      > $text =~ s/\$(\w+)/$user_defs{$1}/g;
      >
      >
      >
      >
      > --
      > Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503
      777 0095
      > <merlyn@...> <URL:http://www.stonehenge.com/merlyn/
      <http://www.stonehenge.com/merlyn/> >
      > Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
      > See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl
      training!
      >


      ______________________________________________________________________
      This email has been scanned by the MessageLabs Email Security System.
      For more information please visit http://www.messagelabs.com/email
      ______________________________________________________________________



      NOTICE OF CONFIDENTIALITY: Information included in and/or attached to this electronic mail transmission may be confidential. This electronic mail transmission is intended for the addressee(s) only. Any unauthorized disclosure, reproduction, or distribution of, and/or any unauthorized action taken in reliance on the information in this electronic mail is prohibited. If you believe that you have received this electronic mail transmission in error, please notify the sender by reply transmission, or contact helpdesk@..., and delete the message without copying or disclosing it.

      ______________________________________________________________________
      This email has been scanned by the MessageLabs Email Security System.
      For more information please visit http://www.messagelabs.com/email
      ______________________________________________________________________

      [Non-text portions of this message have been removed]
    • Jenda Krynicky
      From: computerdribble ... Compile each test just once and run it repeatedly. Before opening the log file, but after reading the
      Message 2 of 15 , Jan 8, 2008
        From: "computerdribble" <computerdribble@...>
        > The "specific things" are stored in a mysql database as a regex expression
        > Example:
        >
        > ( ( $field =~ /blah/) and ($field =~ /that/) )
        >
        > I open the MySql database, no problem, load the contents into arrays,
        > no problem.
        >
        > Open the log file (approx 1.48megs in size) for reading only.
        >
        > I have a logic loop
        >
        > grab a line from the log file
        >
        > run all regex expressions against the log line grabbed. This could be
        > from 1 to 500 different tests.
        >
        >
        > its done in a EVAL statement.
        >
        > similar to:
        >
        > if ( eval $array[$vartoarraycontents] )

        Compile each test just once and run it repeatedly.
        Before opening the log file, but after reading the expressions do
        something like

        for my $test (@array) {
        $test = eval 'sub { ' . $test . '}'
        or die "Failed to compile '$test' : $@\n";
        }

        and then while looping through the logfile use

        if ($array[$vartoarraycontents]->()) {

        There is a possible catch though! The variables you plan to make
        available to the tests must be declared before the tests are compiled
        by those eval('sub {'...) statements!

        Jenda
        ===== Jenda@... === http://Jenda.Krynicky.cz =====
        When it comes to wine, women and song, wizards are allowed
        to get drunk and croon as much as they like.
        -- Terry Pratchett in Sourcery
      • Computer-Dribble
        Thank you! One problem I ran into was the EVAL was killing 4 gigs of memeory on large log files. At least I traced it back to that. This is why I wanted to
        Message 3 of 15 , Jan 9, 2008
          Thank you!


          One problem I ran into was the EVAL was killing 4 gigs
          of memeory on large log files. At least I traced it
          back to that.

          This is why I wanted to avoid EVAL completely and do
          some macro expansion.

          thanks again!!

          Joe.

          --- Jenda Krynicky <Jenda@...> wrote:

          > From: "computerdribble" <computerdribble@...>
          > > The "specific things" are stored in a mysql
          > database as a regex expression
          > > Example:
          > >
          > > ( ( $field =~ /blah/) and ($field =~ /that/) )
          > >
          > > I open the MySql database, no problem, load the
          > contents into arrays,
          > > no problem.
          > >
          > > Open the log file (approx 1.48megs in size) for
          > reading only.
          > >
          > > I have a logic loop
          > >
          > > grab a line from the log file
          > >
          > > run all regex expressions against the log line
          > grabbed. This could be
          > > from 1 to 500 different tests.
          > >
          > >
          > > its done in a EVAL statement.
          > >
          > > similar to:
          > >
          > > if ( eval $array[$vartoarraycontents] )
          >
          > Compile each test just once and run it repeatedly.
          > Before opening the log file, but after reading the
          > expressions do
          > something like
          >
          > for my $test (@array) {
          > $test = eval 'sub { ' . $test . '}'
          > or die "Failed to compile '$test' : $@\n";
          > }
          >
          > and then while looping through the logfile use
          >
          > if ($array[$vartoarraycontents]->()) {
          >
          > There is a possible catch though! The variables you
          > plan to make
          > available to the tests must be declared before the
          > tests are compiled
          > by those eval('sub {'...) statements!
          >
          > Jenda
          > ===== Jenda@... === http://Jenda.Krynicky.cz
          > =====
          > When it comes to wine, women and song, wizards are
          > allowed
          > to get drunk and croon as much as they like.
          > -- Terry Pratchett in Sourcery
          >
          >
          >
          > Unsubscribing info is here:
          > http://help.yahoo.com/help/us/groups/groups-32.html
          > Yahoo! Groups Links
          >
          >
          > mailto:perl-beginner-fullfeatured@yahoogroups.com
          >
          >
          >



          ____________________________________________________________________________________
          Never miss a thing. Make Yahoo your home page.
          http://www.yahoo.com/r/hs
        • Jenda Krynicky
          - No I don t. And others don t either. - But you do not need to read what I wrote originaly and what you replied last time, you remember it, don t you? -
          Message 4 of 15 , Jan 9, 2008
            - No I don't. And others don't either.
            - But you do not need to read what I wrote originaly and what you
            replied last time, you remember it, don't you?
            - Because it's all backwards.
            - Why is that?
            - Because it's hard to read.
            - Why?
            - Please do not top-post!

            Since the proposed change decreases the number of eval""s from
            (number of tests)*(number of lines in logfile)
            to just
            (number of tests)
            any leaks in eval"" should be irrelevant.

            Jenda

            From: Computer-Dribble <computerdribble@...>
            > One problem I ran into was the EVAL was killing 4 gigs
            > of memeory on large log files. At least I traced it
            > back to that.
            >
            > This is why I wanted to avoid EVAL completely and do
            > some macro expansion.
            >
            > thanks again!!
            >
            > Joe.
            >
            > --- Jenda Krynicky <Jenda@...> wrote:
            >
            > > From: "computerdribble" <computerdribble@...>
            > > > The "specific things" are stored in a mysql
            > > database as a regex expression
            > > > Example:
            > > >
            > > > ( ( $field =~ /blah/) and ($field =~ /that/) )
            > > >
            > > > I open the MySql database, no problem, load the
            > > contents into arrays,
            > > > no problem.
            > > >
            > > > Open the log file (approx 1.48megs in size) for
            > > reading only.
            > > >
            > > > I have a logic loop
            > > >
            > > > grab a line from the log file
            > > >
            > > > run all regex expressions against the log line
            > > grabbed. This could be
            > > > from 1 to 500 different tests.
            > > >
            > > >
            > > > its done in a EVAL statement.
            > > >
            > > > similar to:
            > > >
            > > > if ( eval $array[$vartoarraycontents] )
            > >
            > > Compile each test just once and run it repeatedly.
            > > Before opening the log file, but after reading the
            > > expressions do
            > > something like
            > >
            > > for my $test (@array) {
            > > $test = eval 'sub { ' . $test . '}'
            > > or die "Failed to compile '$test' : $@\n";
            > > }
            > >
            > > and then while looping through the logfile use
            > >
            > > if ($array[$vartoarraycontents]->()) {
            > >
            > > There is a possible catch though! The variables you
            > > plan to make
            > > available to the tests must be declared before the
            > > tests are compiled
            > > by those eval('sub {'...) statements!
            > >
            > > Jenda
            > > ===== Jenda@... === http://Jenda.Krynicky.cz
            > > =====
            > > When it comes to wine, women and song, wizards are
            > > allowed
            > > to get drunk and croon as much as they like.
            > > -- Terry Pratchett in Sourcery
            > >
            > >
            > >
            > > Unsubscribing info is here:
            > > http://help.yahoo.com/help/us/groups/groups-32.html
            > > Yahoo! Groups Links
            > >
            > >
            > > mailto:perl-beginner-fullfeatured@yahoogroups.com
            > >
            > >
            > >
            >
            >
            >
            > ____________________________________________________________________________________
            > Never miss a thing. Make Yahoo your home page.
            > http://www.yahoo.com/r/hs
            >
            >
            > Unsubscribing info is here: http://help.yahoo.com/help/us/groups/groups-32.html
            > Yahoo! Groups Links
            >
            >
            >
            >


            ===== Jenda@... === http://Jenda.Krynicky.cz =====
            When it comes to wine, women and song, wizards are allowed
            to get drunk and croon as much as they like.
            -- Terry Pratchett in Sourcery
          • computerdribble
            I humbly thank ALL who have read or responded. Can you, with all of the examples to date... do away with the EVAL of the variable that contains the testing
            Message 5 of 15 , Jan 9, 2008
              I humbly thank ALL who have read or responded.

              Can you, with all of the examples to date...
              do away with the EVAL of the variable that contains the testing
              information and just do a straight macro expansion ?

              $this = "that"
              $that ="this"

              $test = ' ( ( "that" =~ /at/) and ( "this" =~ /at/ ) ) '

              something like @{[ $$test ] }


              I'm sorry to have ruffled so many feathers, but just because one has
              20 years of experience in a language does not mean they should look
              down on someone TRYING to learn it. Much less understand it andits
              quirkiness.

              Usually, a pre-madonna is a target in the computer field, remember that.

              I find that perl is never as straight forward as one thinks. It takes
              a great deal of trial and error to get it to do what you want it to do
              .. for a beginner, its frustrating. For someone with 30 years
              experince in other languages, its VERY frustrating. Unless I
              completely dive into the language full time.. I'll never gain the
              knowledge you guys have. But I don't need to. This is a 1 time
              thing. I can go someone else to find the answers BUT... I thought
              that this was a "perl-beginner" group/forum not a "lets pick on
              someone trying to learn the language" area.
              Is this what "knowledge sharing" has degraded to ?

              Most of the "trial and error" is because of the REGEX. Each language,
              has its own way of looking at REGEX and Perl is notorious for its own
              quirks.

              I bow before greater GODs than I in the Perl field and I humbly do so.
              If I had my way, I would have written this in another language
              altogether, but a client wants what a client wants.

              As for "top loading or top responding" Not my fault, blame Yahoo for
              doing that way.. hell blame the whole freaking internet for it,
              because that's just how it is done. Talk to the hand.






              Joe.
            • Jenda Krynicky
              From: computerdribble ... No. You can t. It s as simple as that. If you did not store the expression like this in the database,
              Message 6 of 15 , Jan 9, 2008
                From: "computerdribble" <computerdribble@...>
                > I humbly thank ALL who have read or responded.
                >
                > Can you, with all of the examples to date...
                > do away with the EVAL of the variable that contains the testing
                > information and just do a straight macro expansion ?
                >
                > $this = "that"
                > $that ="this"
                >
                > $test = ' ( ( "that" =~ /at/) and ( "this" =~ /at/ ) ) '
                >
                > something like @{[ $$test ] }

                No. You can't. It's as simple as that.

                If you did not store the expression like this in the database, but
                instead stored the name of the variable and the regexp (and nothing
                else) in separate columns you could do something like

                if ($data{$varname} =~ /$regexp/) {

                where the $varname and $regexp comes from the database. Whether that
                would be enough for you, I don't know.

                Even in this case though it would be better to preprocess the
                regexps:

                foreach my $test (@tests) {
                # $test = [$varname, $regexp]
                eval {
                $test[1] = qr/$test[1]/
                } or die qq{Invalid regexp /$test[1]/ : $@\n};
                }

                This way the regexps are parsed and compiled just once instead of
                each time they are used.

                > I'm sorry to have ruffled so many feathers, but just because one has
                > 20 years of experience in a language does not mean they should look
                > down on someone TRYING to learn it. Much less understand it andits
                > quirkiness.
                >
                > ...

                I think all this was unnecessary. Noone's picking on you.

                > As for "top loading or top responding" Not my fault, blame Yahoo for
                > doing that way.. hell blame the whole freaking internet for it,
                > because that's just how it is done. Talk to the hand.

                The fact that Yahoo (by default?) formats replies one way doesn't
                mean you have to keep it that way. Top-posting is quite OK for
                personal emails to a single recipient. For mailing lists its ... not
                working too well. Always try to make it easy to the people you are
                asking for help, even if it means you spend two more minutes
                formatting the email.

                Jenda
                ===== Jenda@... === http://Jenda.Krynicky.cz =====
                When it comes to wine, women and song, wizards are allowed
                to get drunk and croon as much as they like.
                -- Terry Pratchett in Sourcery
              • computerdribble
                Thank you for responding. ... dang it all! oh well. ... I would love to.. except there are certain instances that what needs to be checked is.. well let me
                Message 7 of 15 , Jan 9, 2008
                  Thank you for responding.


                  --- In perl-beginner@yahoogroups.com, "Jenda Krynicky" <Jenda@...> wrote:
                  >
                  > From: "computerdribble" <computerdribble@...>
                  > > I humbly thank ALL who have read or responded.
                  > >
                  > > Can you, with all of the examples to date...
                  > > do away with the EVAL of the variable that contains the testing
                  > > information and just do a straight macro expansion ?
                  > >
                  > > $this = "that"
                  > > $that ="this"
                  > >
                  > > $test = ' ( ( "that" =~ /at/) and ( "this" =~ /at/ ) ) '
                  > >
                  > > something like @{[ $$test ] }
                  >
                  > No. You can't. It's as simple as that.
                  >


                  dang it all!
                  oh well.


                  > If you did not store the expression like this in the database, but
                  > instead stored the name of the variable and the regexp (and nothing
                  > else) in separate columns you could do something like
                  >
                  > if ($data{$varname} =~ /$regexp/) {
                  >
                  > where the $varname and $regexp comes from the database. Whether that
                  > would be enough for you, I don't know.
                  >
                  > Even in this case though it would be better to preprocess the
                  > regexps:
                  >
                  > foreach my $test (@tests) {
                  > # $test = [$varname, $regexp]
                  > eval {
                  > $test[1] = qr/$test[1]/
                  > } or die qq{Invalid regexp /$test[1]/ : $@\n};
                  > }
                  >


                  I would love to.. except there are certain instances that what needs
                  to be checked is.. well let me put it this way:


                  (( $field =~ /this/) and ( ($field =~ /that/) or ($field1 =~/a/) or
                  $field2 =~/xx/) or... or.... or.....) )

                  actually some of the rules/checks are quite extensive. I **wish**
                  they were easier.. but they are not and will probably be redefined to
                  something more tighter.. looking for VERY specific things.



                  I appreciate all of the help and guidance.


                  joe






                  > This way the regexps are parsed and compiled just once instead of
                  > each time they are used.
                  >
                  > > I'm sorry to have ruffled so many feathers, but just because one has
                  > > 20 years of experience in a language does not mean they should look
                  > > down on someone TRYING to learn it. Much less understand it andits
                  > > quirkiness.
                  > >
                  > > ...
                  >
                  > I think all this was unnecessary. Noone's picking on you.
                  >
                  > > As for "top loading or top responding" Not my fault, blame Yahoo for
                  > > doing that way.. hell blame the whole freaking internet for it,
                  > > because that's just how it is done. Talk to the hand.
                  >
                  > The fact that Yahoo (by default?) formats replies one way doesn't
                  > mean you have to keep it that way. Top-posting is quite OK for
                  > personal emails to a single recipient. For mailing lists its ... not
                  > working too well. Always try to make it easy to the people you are
                  > asking for help, even if it means you spend two more minutes
                  > formatting the email.
                  >
                  > Jenda
                  > ===== Jenda@... === http://Jenda.Krynicky.cz =====
                  > When it comes to wine, women and song, wizards are allowed
                  > to get drunk and croon as much as they like.
                  > -- Terry Pratchett in Sourcery
                  >
                Your message has been successfully submitted and would be delivered to recipients shortly.