On Nov 14, 2003, at 3:47 PM, Brass Tilde wrote:
>> On Friday 14 November 2003 20:03, Brass Tilde wrote:
>> We currently have >6Mb of data files... 30,000 lines so
>> probably 25,000 items
> And you had performace problems with that tiny little thing? One of
> tables in our main database, constantly transacted against, currently
> over 4 million rows. Several others have in excess of a million. That
> doesn't count the ancillary databases used for non-Core operations. No
> primary keys *really* sucks in that environment.
Yep. Serious ones. Go take a look at how we manage things in pcgen
currently. There are no primary keys, and we analyze strings like
"Foo|BAR|fizBot|PREQ:STR>3:Green" with a string tokenizer on "|" and
then several more on ":" to see whether we should show something in
green or bold, then eval the prequisite (stored in string form) to see
if the character meets it, and thus should be show in red.
Last time I checked, showing the feat screen with just the Big 3 and
Monte's ranger requires running something like 85,000 strings through a
bonus checker. I believe that some optimization work has been done
since then, but we have not set up real dependencies between objects,
partially because of a lack of keys, and thus we end up re-doing a lot
of work on tab display.
This technique was not a bad one when we had just a few items, with
relatively simple formulas, but we now have zillions of them. Having
unique ids and objects would make it really, really easy to find
objects by key, so we could then parse the formulas only once and
evaluate them as needed without lots of code knowing the difference.
If all we do is provide a unique id that can be used in maps/sets for
object lookup, and get all the refs hooked up right. we will have done
a good thing. If we also parse the formulas once, we will cause
probably three or four orders of magnitude improvement in
display/calculation speed. As a bonus, we could determine those things
to be drawn in a given font/color and paint those, which can result in
three orders of magnitude improvement in draw time on MacOS X.
(Changing fonts/colors is pricey, as it goes through a compositor.)