Member Avatar for stupidenator

Hello,
I am working on a project and I need to go to multiple news web sites and get articles pertaining to stock numbers. My ideas so far have been to download an RSS file from somewhere like Google Finance, and then extract the links out of there, follow them, get just the article section and then store it into a database. The problem I am seeing is that the sites are structured too differently for me to write something that can accomplish this. I am looking for help on getting a little more advanced with the scraping and wondering if someone could maybe recommend some perl modules that might make this a little easier.

Thanks for the help!
--
Nick

  • 3 Contributors
  • 2 Replies
  • 157 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by trudge
Member Avatar for KevinADC

search CPAN for RSS modules. I have no specific recommendations.

Member Avatar for trudge

WWW::Mechanize is the de facto module for scraping, and other tasks. Beware though if the target site contains JavaScript, as Mechanize will not execute it.

Also see http://www.research.att.com/sw/tools/wsp/

And FEAR::API at CPAN.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.