language agnostic - Burstable Output for Long running scripts -

script

<?php include('time.php'); //time script echo "first 100 users of so<br/>"; for($i=0; $i<100;$i++){ $contents=file_get_contents("https://stackoverflow.com/privileges/user/".$i); preg_match('!<div class="summarycount al">(.+?)</div>!', $contents, $matches); $rep = $matches[1]; echo "<br/>".$i.") ".$rep."<br/>";  include('timetaken.php'); //script outputs time difference } ?> 

output

first 100 users of  0) 0 2.3584280014038 1) 14,436 4.469074010849 2) 875 10.651238918304 3) 2,431 12.991086959839 4) 8,611 15.451638936996 5) 14,988 17.535580873489 6) 0 19.686461925507 7) 0 21.796992063522 8) 218 23.931365013123 9) 2,569 26.419286966324 10) 101 28.540382862091 11) 232 30.755586862564 12) 0 32.960548877716 13) 33,898 35.163224935532 14) 0 37.280658006668 15) 6,388 39.425274848938 16) 143 41.541421890259 17) 14,366 43.655340909958 18) 0 45.771246910095 19) 99 47.882269859314 20) 4,204 49.993322849274 21) 0 52.108762979507 22) 1,517 54.221307039261 23) 411 56.345490932465 24) 103 58.892389059067 fatal error: maximum execution time of 60 seconds exceeded in c:\test.php on line 5 

problems script: 1. page loads after 60 seconds when timesout

i know can add

set_time_limit(500); 

to code , first 100 reputations, result in page load after 120seconds.

how can result in short bursts, data gathered using php or using other language (python, java) or else. before says it, have read best way manage long-running php script?, may possible duplicate , not answer question. my question not completing entire job, displaying results being done.

(please deal tags me)

if sample you're doing, how harvesting data, , storing locally? that'd speed things lot. can periodically reharvest, unless data's time sensitive, isn't want on every page load.

alternatively, i'd consider moving logic on client. here's 2 approaches:

1) have php process takes parameter record start with. make ajax calls process each 1 grabbing next set of records, , adding result dom. (jquery's 1 way this.) depending on needs, , testing shows, have callback each launch next request, or launch several gets @ time.

2) skip php entirely, , javascript in browser. after all, you're loading , parsing html (although might have deal cross-domain issues).