39Re: [AI-GEOSTATS: MSE to compare different methods]
- Jan 7, 2001Dear Mercedes,
doing k fold cross validation (taking out X % of the samples) will not give
you any reliable results unless you repeat the operation several times. Taking
out 15% of the samples one time only will give you an MSE that will depend
strongly on the data you have removed. Has the selection of the 15% been made
randomly? You may get a strong bias if the 15% of the samples have been taken
in one region in particular or if you have taken out extreme values only. At
this stage, I would trust more the results obtained by standard cross
validation (leave one out method).
I didn�t check your previous mail but if you have few samples only,
k-fold cross validation won�t help you much.
If you have many samples, then you should repeat the procedure at least 10
times to be sure that the way you have extracted the data has not influenced
too much the results.
Also, if you have a phenomenon that fluctuates at different scales, you may
have removed the short scale effect by taking out only few samples (15% is not
My suggestion is the following: it is time consuming but might be worth the
effort. The idea is to take out an increasing number of samples (10, 20, 30,
40, 50, 60, ...,X%) of samples, this 10 times, and see how the average MSE
evolves. You may find out that methods A & B work better than C & D when only
few samples are removed and that C & D give better results than A & B when
more than 40% of the samples have been removed. This would mean that C & D
describe better the general trend of the phenomenon while A & B are more
sensitive to the local structures (since you have more dense data).
If you don�t have the time to proceed in such a way, you should use standard
cross validation only and investigate the regions/samples where you have the
Just few thoughts.
"Berterretche, Mercedes" <Mercedes.Berterretche@...> wrote:
>any useful responses to your questions.
> I would like to thank Benjamin Warr for his siggestion about doing
> difference images instead of global measures as MSE.
> I'm confused because crossvalidation MSE (taking one sample out and
> recalculating) and validation MSE (taking 15 percent of the samples out and
> recalculating) are giving me opposite results. The validation method would
> allows me to compare kriging vs cokriging vs Kriging with an external drift
> vs regression , but I don't know if I can trust the results at this point.
> Does anybody have any input about this?
> Thanks in advance,
> Mercedes Berterretche
> * To post a message to the list, send it to ai-geostats@...
> * As a general service to the users, please remember to post a summary of
> * To unsubscribe, send an email to majordomo@... with no subject and"unsubscribe ai-geostats" followed by "end" on the next line in the message
body. DO NOT SEND Subscribe/Unsubscribe requests to the list
> * Support to the list is provided at http://www.ai-geostats.orgGregoire Dubois (Ph.D.)
Institute of Mineralogy and Petrography
Dept. of Earth Sciences
University of Lausanne
Get free email and a permanent address at http://www.netaddress.com/?N=1
* To post a message to the list, send it to ai-geostats@...
* As a general service to the users, please remember to post a summary of any useful responses to your questions.
* To unsubscribe, send an email to majordomo@... with no subject and "unsubscribe ai-geostats" followed by "end" on the next line in the message body. DO NOT SEND Subscribe/Unsubscribe requests to the list
* Support to the list is provided at http://www.ai-geostats.org
- Next post in topic >>