Member Avatar for kooben

Hi

I intend over the next few months to learnt Java with the purpose of building my own simple web crawler/spider. I have seen a few open source spiders but would like to build my own if possible.

What I would like to ask is how would I go about learning java and also would the building of a simple spider be very hard?

My requirements of the spider are as follows:

Go to the entered URL and gather all content from the site
Collect link structure

The app I am developing will need to be able to build a structured sitemap of the specified URL.

One final note is how would I go about building a browser add-on? What languages can they be built in and which browser is best/easiest to develop for?

Thanks

  • 11 Contributors
  • 12 Replies
  • 323 Views
  • 4 Years Discussion Span
  • Latest Post Latest Post by peter_budo
Member Avatar for Black Knight

I have built a java web crawler/spider before with a front end resembling google for a previous uni project and I would say it is a moderate program to try and do, not overly difficult but a definate challenge for a new java coder.

Some of the main bits you will need to learn to do this is iostreams to read the urls in and JDBC so that you can store the data(you could do it by reading into an array/vector but i wouldnt recommend it as it would eat memory).

There is loads on the web about spider methods and algorithms like word ranking etc but i am sure you have already read up about how they work.

It is probobly quite a good project as you could make it on the command line and then redo it with a gui later if you wanted to.

As for browser plugins I would probobly go for a firefox plugin but then again why stop at a search engine, why not build your own browser too. :mrgreen:

Member Avatar for server_crash

I think the java.sun site had a tutorial on creating one of these. This is actually the next project I want to take up!

Member Avatar for Dark Master

hi black knight, i m also building a web crawler in java as a project work.can u giude me?i m new to java.

Member Avatar for shubh_9797

can any one help me out i want to build a web crawler

commented: Unnecessarily hijacking an old thread. -1
commented: 1. hijacking 's never fun 2. what's stopping you? -1
Member Avatar for Ezzaral

Only if you start a new thread for your request and demonstrate that you have made some effort on your own.

Member Avatar for shelley7753

that link doesn't work...... and i can't find wizard solutions website, to try to join or whatever... how do you get to wizard solutions

Member Avatar for Ezzaral

The post is over four years old. Not everything on the web persists forever.

Member Avatar for shelley7753

actually, i wasn't interested in that topic anymore

Member Avatar for akash210

hi i am akash & i am designing my web crawler in JAVA, can you please send me the code....my id is <EMAIL SNIPPED>....thanks

Edited by peter_budo because: Keep It On The Site - Do not post asking for an answer to be sent to you via email or PM.
Member Avatar for peter_budo

hi i am akash & i am designing my web crawler in JAVA, can you please send me the code....my id is <EMAIL SNIPPED>....thanks

Here is the code

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.