Member Avatar for jen140

Hello all.
I have a small problem.
I have a crewler, that gets into one website and extract some info from it, and writes the exctracted content to file.
The only prob, that in 3~4hours of work, thise script uses 1 gig of ram.

for(my $increment = 1; $increment <= 999999; $increment++){
my $funky = "http://www.website.com/page?id=".$increment;
print $increment."\n";
my $content = get($funky);

THere is the part of the code, i tryied the undef function(at the end of for cycle) but the sam ram was used after 3~4hours of crewling.
So i would like to know, if is it possible to clean the memory at the end of the for cycle, and how .
Thanks in advance.

  • 3 Contributors
  • 5 Replies
  • 105 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by lordspace

Dani AI

Generated

: long-running crawlers commonly hit two separate things — real leaks (references kept alive by your code or modules) and allocator behavior (Perl/free() rarely returns memory to the OS even after you undef things). 's and 's comments point at useful tactics, but they won't always find hidden references or module-level caches.

Useful diagnostics (start here)

  • Use Devel::Size to measure which big structures grow: it shows bytes used by a variable (Devel::Size).
  • Use Devel::Cycle to detect circular references that prevent garbage collection (Devel::Cycle).
  • Inspect modules you use: HTTP clients, cookie jars, connection caches and SSL layers can retain state across requests. Check LWP::UserAgent and similar docs for caching behavior (LWP::UserAgent).

Practical remediation patterns

  • Break the process into short-lived workers (batch + fork) so the OS reclaims memory when a child exits. Example pattern:
    # parent spawns a child per batch; child does work then exits
my $batch = 500;
for (my $start = 1; $start <= $max; $start += $batch) {
my $pid = fork();
die "fork: $!" unless defined $pid;
if ($pid == 0) {
  # child: fetch start..start+$batch-1, write results to disk/DB, then exit
  exit(0);
}
waitpid($pid, 0);
}
  • Avoid accumulating results in memory: stream writes to files or insert to a DB (SQLite/DBI) instead of pushing into large arrays/hashes.
  • Replace heavy clients with lighter ones for long runs (for example, try HTTP::Tiny) and be explicit about clearing cookie/connection caches when available (HTTP::Tiny).

Short checklist

  • Add periodic self-checks of RSS (ps/pmap) to spot when growth accelerates.
  • Run Devel::Size/Devel::Cycle on a reproducer to find the leak source.
  • If a module holds the leak, either reinitialize/clear its caches or isolate it in a short-lived child process.
  • Use Scalar::Util::weaken to break intentional circular refs when appropriate (Scalar::Util).

These steps let you locate whether the growth is your data structures or module-level caching/allocator behavior and choose the right fix (fix leak vs. restart worker).

Recommended Answers

All 5 Replies

Member Avatar for lordspace

Why don't you call an external binary e.g. curl to get the content.
You might want to add some headers (at least user agent) so the site doesn't block you by detecting the user agent string.
Executing an external binary will be a separate process and the memory will be released by the OS.

Member Avatar for jen140

Ok ,thanks for a nice ideia, i am going to try it .But still , for the future would like to know how to manage memory in perl.

Member Avatar for KevinADC

Normally in perl you don't worry about memory management. But instead of undef you can assign the variable an empty list/string and see if that helps free up memory.

$var = '';
@var = ();
%var = ();

Member Avatar for jen140

It helped a bit, at start . But in 20~30 minutes the memory used is same =( . Or possibly im missing some variable. Gonna check. Is there a way to print all the variable names ?

Member Avatar for lordspace

Obviously perl doesn't release the memory until the script exists.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.