transition from commercial testing tool to perl
- I'm putting together a plan to transition our Visual Test tools to Perl.
I'm guessing / hoping that others of you have done something similar.
Could you share a brief experience and some pointers for one about to do it?
Some other questions that come to mind are:
- how large will each stand alone piece be in lines of code / functionality?
- how do your parts talk to each other?
- user input, error detection, crash detection, performance, memory
- what is the most basic input that tells what actions are done?
- currently we use text files ( grids ) with basic language
- one common engine for every test
- many grids, used to create unique tests
- what balance have you found between reusability and simple copy/paste of
- I want to get something working quickly, but slow enough to avoid
- how many things do you have running in parallel during testing?
- currently we have one engine giving input, and a crash monitor
- how much importance do you place on logging stats to some network archive
while things are running?
- do you push out data from test targets to a network share...
- or pull it from the test target?
- when a failure happens do you auto archive and reboot and restart?
- how important is this?
This message and any included attachments are from Siemens Medical Solutions
USA, Inc. and are intended only for the addressee(s).
The information contained herein may include trade secrets or privileged or
otherwise confidential information. Unauthorized review, forwarding, printing,
copying, distributing, or using such information is strictly prohibited and may
be unlawful. If you received this message in error, or have reason to believe
you are not authorized to receive it, please promptly delete this message and
notify the sender by e-mail with a copy to Central.SecurityOffice@...
[Non-text portions of this message have been removed]
- Boy, you are brave!!!
I am a strong supporter of free software and rarely resist the
temptation to criticize commercial crap. But...
You have to be aware, that Win32::GuiTest is a relatively young
module, with increasing, but still not that big users community.
I am currently trying to use Win32::GuiTest to automate testing in
company I work in and these are my observations on missing features
(which may also be not present in commercial tools):
* There is no framework for writing easy to maintain complex test
scripts. I am working on such a framework, I started to use it but it
is still in a state of limited use for others (limited documentation,
* There is no support for custom controls (seems to be a problem in
any tool, even commercial). Example of such a control is WinWidget
* Documentation for Win32::GuiTest apart from obscure reference and
tutorial about basics is almost not existent.
* No recording facility.
In my work, I am using WinSpy to inspect windows. And I am automating
application written in VC++ (not Java, not .NET).
To put it short. If you are good at programming and are willing/have
time to learn about Win32 windowing mechanics - go for it.
If you are not that familiar with programming and do not have a
skilled programmer available, think twice or even 3 times before you
fully go to Win32::GuiTest
Now let me tell you what are advantages (from my point):
* It uses a mature, well known, extensible language (perl)
not YAPL (Yet Another Propietary Language)
* I know the internals so I can expand it to adjust to custom controls
(this require some knowledge though, and acquiring it takes some time)
* It is free.
* I believe that after having some success stories, this module may
become really popular and well supported.
PS: Let us know what is your decision.
--- In firstname.lastname@example.org, Coleman Tom <tom.coleman@s...>
> I'm putting together a plan to transition our Visual Test tools to
> I'm guessing / hoping that others of you have done somethingsimilar.
>to do it?
> Could you share a brief experience and some pointers for one about
> Some other questions that come to mind are:
- Once upon a time I worked at a large computer manufacturing company;
where due (primarily) to the cost of the proprietary tool (~45k/qtr)
we needed to look at something else... We evaluated several
tools... Our requirements generally centered on:
* LOW/NO COST
- ActiveState Perl and Win32::GuiTest were free. Other tools were
at least several thousand.
* FUNCTIONALITY (able to interact/test standard GUI applications,
and have a script language we can use for various other tasks)
- While other tools had a lot of nifty functionality,
Win32::GuiTest suited our needs. Being open-source, we knew that we
could always enhance it in-house or pay someone else to do that for
us, still much cheaper then paying a re-occuring license/support fee.
- We leveraged Perl scripting to drive our test suite.
* EASE OF USE
- Anyone with much experience with current automated tools should
know that, relying heavily on a "script recorder" isn't the way to
go and only by diving into the test language and understanding what
it and the application is doing will you get the full benefit... We
knew that Perl being a standard language was going to be easy enough
to learn. The commands in Win32::GuiTest were very similar to
proprietary tools. We used the freeware WinSpy to obtain the window
information that we needed.
* STABILITY (had to execute on over 6,000 varying system
configurations daily, under varying Windows OSs, etc.)
- Perl and Win32::GuiTest were checked to ensure their stability
in our particular environment. All went well. In an indirect way,
Perl even helped us find one or two bad hard drives.
* SPEED (we did NOT have manufacturing time to waste for tools that
had to initialize their "engines" for many seconds, etc.).
- Win32::GuiTest was much faster then our previous tool in loading
(no "engine" needed to be loaded) and in script execution. A close
second on script execution speed was a proprietary tool with a
* SMALL/NON-INTRUSIVE IMPRINT (we had to download and remove the
tool from each system, so it couldn't be of a huge size and we
didn't want unnecessary Windows registry cruft lingering here and
- Perl/Win32::GuiTest were half the size of our previous tool and
we could deliver it to the system in flat-file format without having
to register/install this and that.
There is more to the story, but in the end our project was completed
on time, we eliminated a ~45k quarterly fee, speed up manufacturing
by at least 3 minutes per system, were able to add a few additional
checks/tests that were needed, increased quality, gave our team
something to put on their resume, etc.
> - how large will each stand alone piece be in lines of code /functionality?
With our use of Perl/Win32::GuiTest (mostly scripting, several
scripts for GUI interacting/testing), we put a 500 line cap on each
script and none of the GUI interacting/testing scripts came close to
that limit... You could probably have the scripts seperated by
which parent window they work with, depends on your project.
> - how do your parts talk to each other?memory> usage, etc.
> - user input, error detection, crash detection, performance,
Error and crash detection were handled in the individual script,
whether a window didn't come up (WaitWindow()), button didn't get
disabled (IsWindowEnabled) when it was supposed to, etc. We used
syntax like "IsWindowEnabled($TheWin) or CriticalError(....);"...
CriticalError() was the sort of function we put into a "result
module" that we also whipped up in Perl. We had one master script
that used a data table and a "test case" file to determine which
tests to run. The master script would then call these test scripts
using Perl's do() (as discussed elsewhere in this message board).
These scripts would all use the commands in the Perl module we wrote
to report their findings.. After testing, the master script called
one final script to log results into a database and (if applicable)
report results back to the operator on the manufacturing floor.
> - what is the most basic input that tells what actions are done?As discussed briefly above.
> - currently we use text files ( grids ) with basic language
> - one common engine for every test
> - many grids, used to create unique tests
> - what balance have you found between reusability and simple
> code?Functions that were generic enough to be reusable were placed into a
>- I want to get something working quickly, but slow enough to avoidOur project was phased, so we were going to start out doing
> spagetti code
> - how many things do you have running in parallel during testing?
> - currently we have one engine giving input, and a crash >
sequential testing and work towards parallel testing at a later
point. I was recruited by another company, so I never worked on
that part. The test scripts were quite robust, so it would have
primarily been just a simple change to the master script
("engine"). Our data table could have had a flag indicating which
scripts won't work in parallel mode.
> - how much importance do you place on logging stats to somenetwork archive
> while things are running?We recorded results at the end of testing. If we needed to manage
test systems centrally (in real-time), we would have recorded
results on the fly... Our results script had a recovery feature in
case the network went down, etc.
>- do you push out data from test targets to a network share...We pushed from the test systems to the database server, as only they
>- or pull it from the test target?
knew when they were done.
>- when a failure happens do you auto archive and reboot and restart?Yes. We wanted to always record problems. We even had a
> - how important is this?
placeholder in the database in case a script failed to execute (as
someone might have deleted it).
Ok I'm done rambling for now. :-)