perl的錯誤「內存不足」的glibc檢測 - 如何解決

Question

我得到這個瘋狂的消息：

Out of memory! 
*** glibc detected *** perl: double free or corruption (!prev): 0x00000003f8f89690 *** 
======= Backtrace: ========= 
/lib64/libc.so.6[0x372bc75916] 
/lib64/libc.so.6[0x372bc78443] 
/usr/lib64/perl5/CORE/libperl.so(Perl_pregfree+0xd1)[0x3c5da74581] 
/usr/lib64/perl5/CORE/libperl.so(Perl_op_clear+0xeb)[0x3c5da3829b] 
/usr/lib64/perl5/CORE/libperl.so(Perl_op_free+0x15c)[0x3c5da3670c] 
/usr/lib64/perl5/CORE/libperl.so(Perl_op_free+0xbc)[0x3c5da3666c] 
======= Memory map: ======== 
00400000-00401000 r-xp 00000000 08:01 2229974       /usr/bin/perl 
00601000-00603000 rw-p 00001000 08:01 2229974       /usr/bin/perl 
0146c000-3f90ce000 rw-p 00000000 00:00 0         [heap] 
372b800000-372b820000 r-xp 00000000 08:01 4587522      /lib64/ld-2.12.so 
...(alot of these).... 
=>> PBS: job killed: vmem 17400672256 exceeded limit 17179869184 

當我跑了，我已經在之前（小巧的文件5-10GB）多次使用的腳本較大的文件（50GB）。我用Google搜索周圍，真的不明白我的代碼的問題可能是什麼有關這些問題的答案所以這裏是我的代碼：

my ($ethnicity) = @ARGV; 
my @map; 

open ($fh, "<", "/path/$filename") or die ("Unable to open input file: $!"); 

open ($fh4, "<", "/path/$filename")  or die ("Unable to open input file: $!"); 
my @ID = <$fh4>; 

open ($fh2, ">", "/path/$filename") or die ("Unable to open output file: $!"); 
open ($fh3, ">", "/path/$filename") or die ("Unable to open output file: $!"); 


my $i=0; 
my $j=0; 

print "The .ped and .map files are being generated\n"; 

while( my $line = <$fh>){ 
if ($.<3){ 
    $map[$j] = $line; 
    $j++;} 
else{ 
    my $temp = $line; 
    $temp =~ m/(\d*) (.*)/; 
    chomp($temp); 
    my $ID = $1; 
    print "$ID- "; 
    my $geno = $2; 
    $geno =~ s /([ATCG])([ATCG])/$1 $2/g; 

    $temp = $ID[$i]; 
    $temp =~ m/(\d*) (1 0 0 [012] [012])/; 
    print "$1\n"; 
    if($ID eq $1){ 
      print $fh2 "$ID $2 $geno\n"; 
      $i++;} 
    else{next;} 
    } 
} 

close $fh; 
close $fh2; 

我做了什麼錯？我認爲我編碼它一行一行地查看，並逐行寫入文件以節省內存。

Answer 1

您正在將整個文件打開爲$fh4加載到內存中，但沒有理由。

my $line_from_fh4 = <$fh4>; 
my ($id_from_fh4, $rest_from_fh4) = $line_from_fh4 =~ m/(\d*) (1 0 0 [012] [012])/; 

while(my $line = <$fh>) { 
    if ($.<3) { 
     push @map, $line; 
     next; 
    } 

    chomp($line); 

    my ($ID, $geno) = $line =~ m/(\d*) (.*)/; 
    $geno =~ s/[ATCG]\K(?=[ATCG])/ /g; 

    print "$ID- $id_from_fh4\n"; 

    if ($ID eq $id_from_fh4){ 
     print $fh2 "$ID $rest_from_fh4 $geno\n"; 
     $line_from_fh4 = <$fh4>; 
     ($id_from_fh4, $rest_from_fh4) = $line_from_fh4 =~ m/(\d*) (1 0 0 [012] [012])/; 
    } 
}