regex - How can I extract a pattern from all files in a directory, using Perl? -

i running command returns 96 .txt files each hour of particular date. gives me 24*96 files 1 day in directory. aim extract data 4 months result in 30*24*96*4 files in directory.

after data need extract "pattern" each of files , display output.

1) below script 1 day date hardcoded in script 2) need make work days in month , need run june october 3) data huge , disk run out of space don't want create these many files instead want grep on fly , 1 output file .

how can efficiently ?

my shell script looks this

for r1 in {0..9};   s1 in {0..95};        echo $r1 $s1        curl  -h "accept-encoding: gzip" "http://someservice.com/getvalue?count=96&data=$s1&fields=hittype,querystring,pathinfo" | zcat > 20101008-mydata-$r1-$s1.txt   done done 

• this returns files need.
• after that, extract url pattern each of file grep "test/link/link2" * | grep category > 1. output

you can use awk command urls

awk -vrs="</a>" '/href/&&/test.*link2/&&/category/{gsub(/.*<a.*href=\"|\".*/,"");print}' file