Loading ...
Sorry, an error occurred while loading the content.
 

Perl hashing speedup ?

Expand Messages
  • Sujitkumar Bade
    Hi, This is regarding perl hashing. I am creating a 3 level hash (actually 2 such hashes at a time) with approx. 2.6 million entries. Regarding length of each
    Message 1 of 2 , Sep 4, 2006
      Hi,

      This is regarding perl hashing. I am creating a 3 level hash (actually
      2 such hashes at a time) with approx. 2.6 million entries.

      Regarding length of each key;

      Level 1 key length – Approx 40 chars
      Level 2 Key length – Approx 12 chars
      Level 3 key length – Approx 40 chars

      Now this hash creation process itself is taking around 3.5 to 4 hrs for
      me.

      Is there any way to improve per default hashing function for speed-up
      (or some better way to approach this requirement)? Any URL's which
      could be helpful on same are welcomed -:)

      Please find some stats (after the message) when this entire hashing was
      going on. This is amount of time taken (in seconds) to hash each of
      10000 entries.

      In case you need any additional information about memory usage,
      Memory used by process – 26 MB (Approx)
      Virtual memory usage – 720 MB (Approx)
      RAM available on machine – 512 MB

      Perl version information-
      perl, v5.6.1 built for MSWin32-x86-multi-thread

      Thanks in advance.

      Thanks
      Sujit

      ------------------------------
      Hashing stats
      Line count Time in seconds
      10000 : Time: 12
      20000 : Time: 9
      30000 : Time: 7
      40000 : Time: 8
      50000 : Time: 12
      60000 : Time: 8
      70000 : Time: 7
      80000 : Time: 8
      90000 : Time: 8
      100000 : Time: 8
      110000 : Time: 12
      120000 : Time: 8
      130000 : Time: 8
      140000 : Time: 9
      150000 : Time: 7
      160000 : Time: 10
      170000 : Time: 8
      180000 : Time: 8
      190000 : Time: 9
      200000 : Time: 10
      210000 : Time: 10
      220000 : Time: 12
      230000 : Time: 8
      240000 : Time: 13
      250000 : Time: 10
      260000 : Time: 9
      270000 : Time: 9
      280000 : Time: 7
      290000 : Time: 9
      300000 : Time: 8
      310000 : Time: 10
      320000 : Time: 7
      330000 : Time: 8
      340000 : Time: 9
      350000 : Time: 10
      360000 : Time: 11
      370000 : Time: 8
      380000 : Time: 9
      390000 : Time: 17
      400000 : Time: 9
      410000 : Time: 11
      420000 : Time: 12
      430000 : Time: 11
      440000 : Time: 9
      450000 : Time: 13
      460000 : Time: 10
      470000 : Time: 13
      480000 : Time: 8
      490000 : Time: 9
      500000 : Time: 11
      510000 : Time: 13
      520000 : Time: 12
      530000 : Time: 15
      540000 : Time: 13
      550000 : Time: 12
      560000 : Time: 17
      570000 : Time: 16
      580000 : Time: 80
      590000 : Time: 35
      600000 : Time: 131
      610000 : Time: 113
      620000 : Time: 433
      630000 : Time: 387
      640000 : Time: 68
      650000 : Time: 46
      660000 : Time: 15
      670000 : Time: 13
      680000 : Time: 13
      690000 : Time: 12
      700000 : Time: 10
      710000 : Time: 13
      720000 : Time: 17
      730000 : Time: 19
      740000 : Time: 25
      750000 : Time: 17
      760000 : Time: 22
      770000 : Time: 13
      780000 : Time: 17
      790000 : Time: 14
      800000 : Time: 14
      810000 : Time: 14
      820000 : Time: 14
      830000 : Time: 18
      840000 : Time: 13
      850000 : Time: 19
      860000 : Time: 14
      870000 : Time: 12
      880000 : Time: 13
      890000 : Time: 13
      900000 : Time: 16
      910000 : Time: 14
      920000 : Time: 17
      930000 : Time: 14
      940000 : Time: 17
      950000 : Time: 18
      960000 : Time: 14
      970000 : Time: 17
      980000 : Time: 17
      990000 : Time: 18
      1000000 : Time: 21
      1010000 : Time: 19
      1020000 : Time: 12
      1030000 : Time: 14
      1040000 : Time: 13
      1050000 : Time: 18
      1060000 : Time: 13
      1070000 : Time: 17
      1080000 : Time: 18
      1090000 : Time: 15
      1100000 : Time: 17
      1110000 : Time: 17
      1120000 : Time: 16
      1130000 : Time: 18
      1140000 : Time: 15
      1150000 : Time: 12
      1160000 : Time: 15
      1170000 : Time: 21
      1180000 : Time: 30
      1190000 : Time: 21
      1200000 : Time: 29
      1210000 : Time: 22
      1220000 : Time: 17
      1230000 : Time: 25
      1240000 : Time: 37
      1250000 : Time: 29
      1260000 : Time: 40
      1270000 : Time: 18
      1280000 : Time: 25
      1290000 : Time: 63
      1300000 : Time: 45
      1310000 : Time: 43
      1320000 : Time: 131
      1330000 : Time: 23
      1340000 : Time: 28
      1350000 : Time: 21
      1360000 : Time: 31
      1370000 : Time: 152
      1380000 : Time: 219
      1390000 : Time: 91
      1400000 : Time: 37
      1410000 : Time: 16
      1420000 : Time: 34
      1430000 : Time: 100
      1440000 : Time: 34
      1450000 : Time: 870
      1460000 : Time: 421
      1470000 : Time: 178
      1480000 : Time: 201
      1490000 : Time: 40
      1500000 : Time: 48
      1510000 : Time: 53
      1520000 : Time: 45
      1530000 : Time: 308
      1540000 : Time: 128
      1550000 : Time: 138
      1560000 : Time: 110
      1570000 : Time: 358
      1580000 : Time: 276
      1590000 : Time: 109
      1600000 : Time: 59
      1610000 : Time: 78
      1620000 : Time: 33
      1630000 : Time: 27
      1640000 : Time: 41
      1650000 : Time: 53
      1660000 : Time: 27
      1670000 : Time: 46
      1680000 : Time: 39
      1690000 : Time: 20
      1700000 : Time: 33
      1710000 : Time: 91
      1720000 : Time: 18
      1730000 : Time: 22
      1740000 : Time: 25
      1750000 : Time: 47
      1760000 : Time: 40
      1770000 : Time: 32
      1780000 : Time: 273
      1790000 : Time: 26
      1800000 : Time: 21
      1810000 : Time: 51
      1820000 : Time: 20
      1830000 : Time: 57
      1840000 : Time: 179
      1850000 : Time: 16
      1860000 : Time: 49
      1870000 : Time: 80
      1880000 : Time: 300
      1890000 : Time: 89
      1900000 : Time: 371
      1910000 : Time: 75
      1920000 : Time: 114
      1930000 : Time: 127
      1940000 : Time: 67
      1950000 : Time: 29
      1960000 : Time: 21
      1970000 : Time: 19
      1980000 : Time: 25
      1990000 : Time: 39
      2000000 : Time: 45
      2010000 : Time: 63
      2020000 : Time: 53
      2030000 : Time: 78
      2040000 : Time: 77
      2050000 : Time: 47
      2060000 : Time: 123
      2070000 : Time: 106
      2080000 : Time: 139
      2090000 : Time: 77
      2100000 : Time: 75
      2110000 : Time: 47
      2120000 : Time: 82
      2130000 : Time: 69
      2140000 : Time: 56
      2150000 : Time: 55
      2160000 : Time: 51
      2170000 : Time: 52
      2180000 : Time: 36
      2190000 : Time: 74
      2200000 : Time: 81
      2210000 : Time: 96
      2220000 : Time: 17
      2230000 : Time: 121
      2240000 : Time: 44
      2250000 : Time: 181
      2260000 : Time: 70
      2270000 : Time: 35
      2280000 : Time: 25
      2290000 : Time: 17
      2300000 : Time: 20
      2310000 : Time: 59
      2320000 : Time: 49
      2330000 : Time: 79
      2340000 : Time: 16
      2350000 : Time: 13
      2360000 : Time: 16
      2370000 : Time: 14
      2380000 : Time: 15
      2390000 : Time: 20
      2400000 : Time: 22
      2410000 : Time: 20
      2420000 : Time: 20
      2430000 : Time: 45
      2440000 : Time: 19
      2450000 : Time: 23
      2460000 : Time: 65
      2470000 : Time: 23
      2480000 : Time: 22
      2490000 : Time: 25
      2500000 : Time: 21
      2510000 : Time: 25
      2520000 : Time: 21
      2530000 : Time: 22
      2540000 : Time: 21
      2550000 : Time: 21
      2560000 : Time: 20
      2570000 : Time: 22
      2580000 : Time: 21
      2590000 : Time: 25
      -------------------------------------------------------
    • merlyn@stonehenge.com
      ... Sujitkumar This is regarding perl hashing. I am creating a 3 level hash (actually Sujitkumar 2 such hashes at a time) with approx. 2.6 million entries.
      Message 2 of 2 , Sep 4, 2006
        >>>>> "Sujitkumar" == Sujitkumar Bade <sujitkumar.bade@...> writes:

        Sujitkumar> This is regarding perl hashing. I am creating a 3 level hash (actually
        Sujitkumar> 2 such hashes at a time) with approx. 2.6 million entries.

        Eww. Ewww ewwww.

        You want a database. Not putting it all inside Perl.

        --
        Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
        <merlyn@...> <URL:http://www.stonehenge.com/merlyn/>
        Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
        See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
      Your message has been successfully submitted and would be delivered to recipients shortly.