Re: Quantitative vs. Qualitative Results
- Hi Julian,
--- In firstname.lastname@example.org, Julian Togelius <julian@...> wrote:
> Therefore, I think we should move to a
> system where all papers are initially submitted to venues with low
> acceptance thresholds (and therefore typically high, but not
> pre-determined, acceptance rates), and years later, when they have been
> public long enough to show their real worth in informing other people's
> research, they could be elected to be part of more "selective" proceedings.
> This would likely help researchers refrain from adding needless
> quantitative results just to please reviewers.
We actually do have this. They are called preprint servers, such as arXiv (www.arxiv.org).
- --- In email@example.com, "rhiever489" wrote:
> For example, let's say we did evolve an agent with high-level intelligence using HyperNEAT, qualitatively showed its performance on some cognitive tasks, and plotted the network as a figure. How would someone follow up on that? They could take our code and re-run it, but if high-level intelligence is indeed a rare evolutionary event, how would they stand a chance of reproducing our result? If they can't reproduce it, how could they build on it from there? Or, if it takes a huge amount of effort to reproduce the event, is it really worth the effort to build on it?I feel like you're missing the forest for the trees. So we succeed in evolving human-level intelligence, and you're actually concerned that nothing good will come of it because the evolutionary chain that led to it can't be reproduced? The entire world would be revolutionized in an instant if that happened. We'd be studying the connectivity, firing patterns, and the artificial genome (which would all be explicitly available in their entirety). We'd learn more about intelligence in a few months than in all of preceding human history. We'd be manipulating the genome to see how it alters the intelligence; we'd be playing with the connectivity to see if we can learn how it works and ultimately augment and harness it. Not to mention that every industry on Earth would be employing it.
And in the meantime while all that is going on you'd be complaining about reproducibility? When you start producing revolutionary stuff, reproducing how it happened matters a lot less.
Anyway, let's try to get to a more fundamental question to pinpoint why you put so much faith in quantification as your compass to future progress:
Do you believe that a quantitative result is any more reliable or requires any less subjective interpretation than a qualitative result to decide both whether it is important and where it might lead?
This question deserves some thought because its answer is not obvious. Once you've given it some thought, if your answer is yes, then why would that be? If the answer is no, then why favor quantitative results?
You can guess my answer: At the end of the day, whether a result is quantitative or qualitative, each of us is left with nothing but our own individual intuition to reveal to us its deepest implications. That is the scary fact of real science. There is ultimately no iron edifice of certain truth for us to grab, for all results and all assertions are fallible no matter the method that yields them (think of Newtonian physics). You must in the end rely upon your own mortal human mind to decide for yourself what truth you believe. For that reason we should be open to all evidence of all types. To quote Feyerabend again:
"There is only one principle that can be defended under all circumstances and in all stages of human development. It is the principle: anything goes."
For me, to believe anything otherwise is dangerously naive.