Loading ...
Sorry, an error occurred while loading the content.

GEOSTATS: large dataset in geographic coordinates

Expand Messages
  • Timothy H. Keitt
    Hi, I would like some advise on tools for doing spatial analysis on two large, multivariate spatial datasets. (I ve checked several of the packages listed in
    Message 1 of 5 , Feb 10, 1997
    • 0 Attachment
      Hi,

      I would like some advise on tools for doing spatial analysis on two
      large, multivariate spatial datasets. (I've checked several of the
      packages listed in the ai-geostats home page, but its not clear if they
      will do what I want.) Both of the datasets are in the form of text
      files and the spatial locations are given in geographic (lat-lon)
      coordinates. There are more than 3,000 points in each set and the
      coverage is most of North America. I would like to do an analysis of
      spatial autocorrelation on both datasets, and an analysis of their
      cross-correlation, i.e., to test the hypothesis that one of the
      datasets is influencing the other.

      The main stumbling block has been calculating the distances among the
      lat-lon coordinate pairs. I have been using a combination of PERL and
      the "geod" program from the PROJ.4 distribution. Unfortunately,
      "geod" is written in such a way that it is extremely difficult to call
      repeatedly from within PERL. (If someone could provide documentation
      for the PROJ.4 library routines, I would consider encapsulating them
      in a perl module.) I can't just dump all the pair-wise comparisons
      and then run them through "geod" because it would require at least 2GB
      to store the intermediate data. (Probably more, I ran out of
      memory/disk space long before it finished.)

      So here are a couple of questions:

      1) If I only need +/-1km precision in my distances, is there an
      alternative to PROJ.4, i.e., a simple formula for calculating the
      geodesic?

      2) Can I transform the data points into some Cartesian coordinate
      system and then use simple linear distances? PROJ.4 has many planar
      projections, but its not clear to me that distances wouldn't become
      distorted over an area the size of North America.

      3) Are there any (free Unix ;-) geostats programs that work in
      geographic coordinates and can process large data sets?

      Finally, are there standard methods to test for spatial dependence of
      one dataset on another?

      Thanks,
      Tim

      --
      *To post a message to the list, send it to ai-geostats@....
      *As a general service to list users, please remember to post a summary
      of any useful responses to your questions.
      *To unsubscribe, send email to majordomo@... with no subject and
      "unsubscribe ai-geostats" in the message body.
      DO NOT SEND Subscribe/Unsubscribe requests to the list!
    • Robert K. Pace
      Tim, I have been running some large spatial autoregressions of up to 73,000 observations using maximum likelihood. I would be happy to send you some of my
      Message 2 of 5 , Feb 10, 1997
      • 0 Attachment
        Tim,

        I have been running some large spatial autoregressions of up to 73,000
        observations using maximum likelihood. I would be happy to send you some of my
        forthcoming articles on this topic (many with Ron Barry). These also look at
        kriging with many observations.

        You often don't need to know all the distances, just the distances to the
        observations which have a direct effect upon the observation itself. Why spend
        your computational time looking at distances to observations 100 miles away if
        all that matters occurs within 1 mile? I have one program in Matlab which can
        come up with a spatial weight matrix for a 73,000 observation dataset overnight.
        I suspect the state-of-the-art in this regard is Anselin and Smirnov's (?)
        linked list method which they published in the Journal of Regional Science in
        1996. All of these methods only store the information of interest, which greatly
        reduces storage requirements.

        Many multivariate techniques have spatial variants. I am most familiar with
        regression, if that suits your tastes.

        Kelley Pace
        U of Alaska

        $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
        From: "Timothy H. Keitt" <tkeitt@...>
        To: ai-geostats@...
        Subject: GEOSTATS: large dataset in geographic coordinates
        Sender: owner-ai-geostats@...
        Precedence: bulk
        X-Administrivia-to: majordomo@...
        X-comment: send subscribe/unsubscribe requests to
        ai-geostats-request@...

        Hi,

        I would like some advise on tools for doing spatial analysis on two
        large, multivariate spatial datasets. (I've checked several of the
        packages listed in the ai-geostats home page, but its not clear if they
        will do what I want.) Both of the datasets are in the form of text
        files and the spatial locations are given in geographic (lat-lon)
        coordinates. There are more than 3,000 points in each set and the
        coverage is most of North America. I would like to do an analysis of
        spatial autocorrelation on both datasets, and an analysis of their
        cross-correlation, i.e., to test the hypothesis that one of the
        datasets is influencing the other.

        The main stumbling block has been calculating the distances among the
        lat-lon coordinate pairs. I have been using a combination of PERL and
        the "geod" program from the PROJ.4 distribution. Unfortunately,
        "geod" is written in such a way that it is extremely difficult to call
        repeatedly from within PERL. (If someone could provide documentation
        for the PROJ.4 library routines, I would consider encapsulating them
        in a perl module.) I can't just dump all the pair-wise comparisons
        and then run them through "geod" because it would require at least 2GB
        to store the intermediate data. (Probably more, I ran out of
        memory/disk space long before it finished.)

        So here are a couple of questions:

        1) If I only need +/-1km precision in my distances, is there an
        alternative to PROJ.4, i.e., a simple formula for calculating the
        geodesic?

        2) Can I transform the data points into some Cartesian coordinate
        system and then use simple linear distances? PROJ.4 has many planar
        projections, but its not clear to me that distances wouldn't become
        distorted over an area the size of North America.

        3) Are there any (free Unix ;-) geostats programs that work in
        geographic coordinates and can process large data sets?

        Finally, are there standard methods to test for spatial dependence of
        one dataset on another?

        Thanks,
        Tim

        --
        *To post a message to the list, send it to ai-geostats@....
        *As a general service to list users, please remember to post a summary
        of any useful responses to your questions.
        *To unsubscribe, send email to majordomo@... with no subject and
        "unsubscribe ai-geostats" in the message body.
        DO NOT SEND Subscribe/Unsubscribe requests to the list!

        --
        *To post a message to the list, send it to ai-geostats@....
        *As a general service to list users, please remember to post a summary
        of any useful responses to your questions.
        *To unsubscribe, send email to majordomo@... with no subject and
        "unsubscribe ai-geostats" in the message body.
        DO NOT SEND Subscribe/Unsubscribe requests to the list!
      • Philippe Aubry
        Timothy H. Keitt wrote : Finally, are there standard methods to test for spatial dependence of one dataset on another? Lyon, France
        Message 3 of 5 , Feb 11, 1997
        • 0 Attachment
          "Timothy H. Keitt" <tkeitt@...> wrote :

          "
          Finally, are there standard methods to test for spatial dependence of
          one dataset on another?"


          Lyon, France



          Bonjour Tim,


          First I apologize for my poor english.

          I suggest you investigate the Mantel's test, a permutation-based method
          which allows to check the relationship between two distance matrices build
          on the same set of locations. However, be careful to make a sufficient
          number of randomizations
          (say, at least 10 000), dont't use the so-called "asymptotic results", and
          check the effect (if any) of the distance you wish to use.


          Sinceres salutations

          Philippe AUBRY

          Laboratoire de Biometrie
          UMR CNRS 5558
          Universite Claude Bernard - Lyon 1
          43 bd. du 11 Novembre 1918
          69622 VILLEURBANNE Cedex
          FRANCE

          private fax number : 04.72.74.47.46

          e-mail : paubry@...-lyon1.fr



          --
          *To post a message to the list, send it to ai-geostats@....
          *As a general service to list users, please remember to post a summary
          of any useful responses to your questions.
          *To unsubscribe, send email to majordomo@... with no subject and
          "unsubscribe ai-geostats" in the message body.
          DO NOT SEND Subscribe/Unsubscribe requests to the list!
        • Alok K. Bohara
          Robert, I would be interested in receiving them too. Thanks. Alok Bohara Department of Economics University of New Mexico Albuquerque, NM 87131 ... -- *To
          Message 4 of 5 , Feb 11, 1997
          • 0 Attachment
            Robert,

            I would be interested in receiving them too. Thanks.

            Alok Bohara
            Department of Economics
            University of New Mexico
            Albuquerque, NM 87131



            On 11 Feb 1997, Robert K. Pace wrote:

            > Date: 11 Feb 97 02:37:09 EST
            > From: Robert K. Pace <73132.1000@...>
            > To: GEOSTATS <ai-geostats@...>
            > Subject: Re: GEOSTATS: large dataset in geographic coordinates
            >
            > Tim,
            >
            > I have been running some large spatial autoregressions of up to 73,000
            > observations using maximum likelihood. I would be happy to send you some of my
            > forthcoming articles on this topic (many with Ron Barry). These also look at
            > kriging with many observations.
            >
            > You often don't need to know all the distances, just the distances to the
            > observations which have a direct effect upon the observation itself. Why spend
            > your computational time looking at distances to observations 100 miles away if
            > all that matters occurs within 1 mile? I have one program in Matlab which can
            > come up with a spatial weight matrix for a 73,000 observation dataset overnight.
            > I suspect the state-of-the-art in this regard is Anselin and Smirnov's (?)
            > linked list method which they published in the Journal of Regional Science in
            > 1996. All of these methods only store the information of interest, which greatly
            > reduces storage requirements.
            >
            > Many multivariate techniques have spatial variants. I am most familiar with
            > regression, if that suits your tastes.
            >
            > Kelley Pace
            > U of Alaska
            >
            > $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
            > From: "Timothy H. Keitt" <tkeitt@...>
            > To: ai-geostats@...
            > Subject: GEOSTATS: large dataset in geographic coordinates
            > Sender: owner-ai-geostats@...
            > Precedence: bulk
            > X-Administrivia-to: majordomo@...
            > X-comment: send subscribe/unsubscribe requests to
            > ai-geostats-request@...
            >
            > Hi,
            >
            > I would like some advise on tools for doing spatial analysis on two
            > large, multivariate spatial datasets. (I've checked several of the
            > packages listed in the ai-geostats home page, but its not clear if they
            > will do what I want.) Both of the datasets are in the form of text
            > files and the spatial locations are given in geographic (lat-lon)
            > coordinates. There are more than 3,000 points in each set and the
            > coverage is most of North America. I would like to do an analysis of
            > spatial autocorrelation on both datasets, and an analysis of their
            > cross-correlation, i.e., to test the hypothesis that one of the
            > datasets is influencing the other.
            >
            > The main stumbling block has been calculating the distances among the
            > lat-lon coordinate pairs. I have been using a combination of PERL and
            > the "geod" program from the PROJ.4 distribution. Unfortunately,
            > "geod" is written in such a way that it is extremely difficult to call
            > repeatedly from within PERL. (If someone could provide documentation
            > for the PROJ.4 library routines, I would consider encapsulating them
            > in a perl module.) I can't just dump all the pair-wise comparisons
            > and then run them through "geod" because it would require at least 2GB
            > to store the intermediate data. (Probably more, I ran out of
            > memory/disk space long before it finished.)
            >
            > So here are a couple of questions:
            >
            > 1) If I only need +/-1km precision in my distances, is there an
            > alternative to PROJ.4, i.e., a simple formula for calculating the
            > geodesic?
            >
            > 2) Can I transform the data points into some Cartesian coordinate
            > system and then use simple linear distances? PROJ.4 has many planar
            > projections, but its not clear to me that distances wouldn't become
            > distorted over an area the size of North America.
            >
            > 3) Are there any (free Unix ;-) geostats programs that work in
            > geographic coordinates and can process large data sets?
            >
            > Finally, are there standard methods to test for spatial dependence of
            > one dataset on another?
            >
            > Thanks,
            > Tim
            >
            > --
            > *To post a message to the list, send it to ai-geostats@....
            > *As a general service to list users, please remember to post a summary
            > of any useful responses to your questions.
            > *To unsubscribe, send email to majordomo@... with no subject and
            > "unsubscribe ai-geostats" in the message body.
            > DO NOT SEND Subscribe/Unsubscribe requests to the list!
            >
            > --
            > *To post a message to the list, send it to ai-geostats@....
            > *As a general service to list users, please remember to post a summary
            > of any useful responses to your questions.
            > *To unsubscribe, send email to majordomo@... with no subject and
            > "unsubscribe ai-geostats" in the message body.
            > DO NOT SEND Subscribe/Unsubscribe requests to the list!
            >
            --
            *To post a message to the list, send it to ai-geostats@....
            *As a general service to list users, please remember to post a summary
            of any useful responses to your questions.
            *To unsubscribe, send email to majordomo@... with no subject and
            "unsubscribe ai-geostats" in the message body.
            DO NOT SEND Subscribe/Unsubscribe requests to the list!
          • Wendelin Reich
            ... This sounds very interesting... can you tell me where I could find your articles, too? -- osv/osl... Wendelin Reich
            Message 5 of 5 , Feb 15, 1997
            • 0 Attachment
              Robert K. Pace wrote:

              > Tim,
              >
              > I have been running some large spatial autoregressions of up to 73,000
              > observations using maximum likelihood. I would be happy to send you some of my
              > forthcoming articles on this topic (many with Ron Barry). These also look at
              > kriging with many observations.

              This sounds very interesting... can you tell me where I could find your
              articles, too?

              --
              osv/osl...

              Wendelin Reich wreich@...
              Dept. of Economic Geography Tel (Job): +49-551-398088
              Geographisches Institut der Tel (priv.): +49-551-35906
              Universitaet Goettingen
              D-37077 Goettingen, Germany
              ----------------------------------------------------------------------
              Mistrust and dislike of sensible reasoning demands mistrust
              and dislike of man.
              Sokrates

              --
              *To post a message to the list, send it to ai-geostats@....
              *As a general service to list users, please remember to post a summary
              of any useful responses to your questions.
              *To unsubscribe, send email to majordomo@... with no subject and
              "unsubscribe ai-geostats" in the message body.
              DO NOT SEND Subscribe/Unsubscribe requests to the list!
            Your message has been successfully submitted and would be delivered to recipients shortly.