Loading ...
Sorry, an error occurred while loading the content.

39Re: [AI-GEOSTATS: MSE to compare different methods]

Expand Messages
  • Gregoire Dubois
    Jan 7, 2001
    • 0 Attachment
      Dear Mercedes,

      doing k fold cross validation (taking out X % of the samples) will not give
      you any reliable results unless you repeat the operation several times. Taking
      out 15% of the samples one time only will give you an MSE that will depend
      strongly on the data you have removed. Has the selection of the 15% been made
      randomly? You may get a strong bias if the 15% of the samples have been taken
      in one region in particular or if you have taken out extreme values only. At
      this stage, I would trust more the results obtained by standard cross
      validation (leave one out method).

      I didn�t check your previous mail but if you have few samples only,
      k-fold cross validation won�t help you much.

      If you have many samples, then you should repeat the procedure at least 10
      times to be sure that the way you have extracted the data has not influenced
      too much the results.
      Also, if you have a phenomenon that fluctuates at different scales, you may
      have removed the short scale effect by taking out only few samples (15% is not
      much).

      My suggestion is the following: it is time consuming but might be worth the
      effort. The idea is to take out an increasing number of samples (10, 20, 30,
      40, 50, 60, ...,X%) of samples, this 10 times, and see how the average MSE
      evolves. You may find out that methods A & B work better than C & D when only
      few samples are removed and that C & D give better results than A & B when
      more than 40% of the samples have been removed. This would mean that C & D
      describe better the general trend of the phenomenon while A & B are more
      sensitive to the local structures (since you have more dense data).

      If you don�t have the time to proceed in such a way, you should use standard
      cross validation only and investigate the regions/samples where you have the
      highest errors.

      Just few thoughts.

      Gregoire

      "Berterretche, Mercedes" <Mercedes.Berterretche@...> wrote:
      >
      > I would like to thank Benjamin Warr for his siggestion about doing
      > difference images instead of global measures as MSE.
      >
      > I'm confused because crossvalidation MSE (taking one sample out and
      > recalculating) and validation MSE (taking 15 percent of the samples out and
      > recalculating) are giving me opposite results. The validation method would
      > allows me to compare kriging vs cokriging vs Kriging with an external drift
      > vs regression , but I don't know if I can trust the results at this point.
      >
      > Does anybody have any input about this?
      > Thanks in advance,
      > Mercedes Berterretche
      >
      > --
      > * To post a message to the list, send it to ai-geostats@...
      > * As a general service to the users, please remember to post a summary of
      any useful responses to your questions.
      > * To unsubscribe, send an email to majordomo@... with no subject and
      "unsubscribe ai-geostats" followed by "end" on the next line in the message
      body. DO NOT SEND Subscribe/Unsubscribe requests to the list
      > * Support to the list is provided at http://www.ai-geostats.org


      Gregoire Dubois (Ph.D.)
      Institute of Mineralogy and Petrography
      Dept. of Earth Sciences
      University of Lausanne
      Switzerland

      http://www.ai-geostats.org

      ____________________________________________________________________
      Get free email and a permanent address at http://www.netaddress.com/?N=1

      --
      * To post a message to the list, send it to ai-geostats@...
      * As a general service to the users, please remember to post a summary of any useful responses to your questions.
      * To unsubscribe, send an email to majordomo@... with no subject and "unsubscribe ai-geostats" followed by "end" on the next line in the message body. DO NOT SEND Subscribe/Unsubscribe requests to the list
      * Support to the list is provided at http://www.ai-geostats.org
    • Show all 2 messages in this topic