Java html reading

[KirkPatrick]

I haven't ever messed with any webpages through java and I'm kind of curious as to how it would work. I have a specific question that perhaps someone will be able to answer.

How would one go about pulling certain information from a forum? To better understand what I am asking, I'll give an example:

Lets say I am surfing this forum and am currently at:

http://www.daniweb.com/forums/forum9.html

Then I click view source, this gives me the source of the current page opened.

Now lets say I want to pull out every title in there. Ex. title=""

After pulling out each title I would want to add it to a text file or something of the sort so I can refer to it later. For the mean time I would probably add it to an arraylist of some sort because the next thing I would want it to do is click next page and do it to the following page.

Can anyone shed some light on how to do such a thing? Or perhaps let me know what I need to read up on to get where I'm wanting to go.

[Ezzaral]

You can start with Reading Text From A URL and Regular Expressions.

[KirkPatrick]

Much appreciated response ezzaral, just one quick question when reading the text from the url, does that automatically read the source of the page?

[santiagozky]

I don't get your question. are you asking how to get all the titles programaticly or how to store the title strings?

if you are asking how to ge tthe titles, I would suggest to read line by line and use a regular expression to get the string you like.

Take a look at the java Pattern class: http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html

You probably will be doing something like title=\"([a-zA-Z0-9])+\" and then get the string from the capturing group. (I haven't done this in a while so the example is probably wrong, but it is the idea.)

I did something like this but I have it at home. I'll rescue it at night to show you.

[KirkPatrick]

I don't get your question. are you asking how to get all the titles programaticly or how to store the title strings?

if you are asking how to ge tthe titles, I would suggest to read line by line and use a regular expression to get the string you like.

Take a look at the java Pattern class: http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html

You probably will be doing something like title=\"([a-zA-Z0-9])+\" and then get the string from the capturing group. (I haven't done this in a while so the example is probably wrong, but it is the idea.)

I did something like this but I have it at home. I'll rescue it at night to show you.

I'll see if I can clear up my question a bit better for your understanding.

What I am wanting to do is create a program that will read a webpages source, grab a specific field, and write what the fields string down in a text file.

So another example would be, you're viewing this forum and lets say there is a field in the source named thread title.

Now in the source you see: thread_title="title1"

A few lines down in the same source you see thread_title again but this time it is: thread_title="title2"

Now lets say there are a hundred of these per page, and I am wanting to get all of them and write them to a text file, once page one is done I want to go to the next page (by clicking the next page button/link) and do the same thing.

So I would want me text file to look like this:

title1
title2
title3
title4
etc...

I hope that clears up what I am wanting to do for you, if you have an example that would be awesome. I tend to do better when reading how things work if I have an example in front of me as well.

So to answer your question, I am looking how to store the title strings :)

I'll be reading on patterns, regular expressions, and reading a url in the mean time. Thanks for the help so far guys

[Ezzaral]

Much appreciated response ezzaral, just one quick question when reading the text from the url, does that automatically read the source of the page?

Give it a quick try and you'd have a definitive answer. :)

[Ezzaral]

(Yes, the HTML page source is just marked up text)

[santiagozky]

this is some code I wrote to make a custom crawler to get images from a site. just change the pattern and the method that stores the string (Bajador is a class that downloads the image in my case) and put everything in a loop that goes through all the urls you wish to examine.

try {
                    URL url = new URL(baseURL+"/index.php?id="+par);
                  BufferedReader br= new BufferedReader( new InputStreamReader(url.openStream()));
                  String line;
                  while((line=br.readLine())!=null){
                      line=line.trim();
                      String pat="<img src=\"/(\\w*).jpg\" alt=\"Picture\"/></div>";
                       Pattern pattern = Pattern.compile(pat);
                       Matcher matcher = pattern.matcher(line);
                       boolean matchFound = matcher.find();
                        if (matchFound) {
                            String nombre=matcher.group(1);
                            String urlI=baseURL+"/file/"+nombre+".jpg";
                            Bajador baj= new Bajador(baseURL,nombre,par);
                            baj.start();
                          //  baja(baseURL,nombre);
                           
                        }
                  }
                } catch (MalformedURLException ex) {
                    Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
                } catch (IOException e) {
                    Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, e);
                }

Hope this help you.

[kvprajapati]

You could use HTML Parser, which a Java library used to parse HTML in either a linear or nested fashion. It is an open source tool and can be found on SourceForge.
you could also useThe Swing HTML Parser.

[KirkPatrick]

Wow you guys have been very helpful! I'm appreciative for each of your posts

(Yes, the HTML page source is just marked up text)

Sorry I had left the other day before getting back to you about it, I had assumed it read straight from the source, but just wanted to make sure. Thanks for confirming that.
----

this is some code I wrote to make a custom crawler to get images from a site. just change the pattern and the method that stores the string (Bajador is a class that downloads the image in my case) and put everything in a loop that goes through all the urls you wish to examine.

try {
                    URL url = new URL(baseURL+"/index.php?id="+par);
                  BufferedReader br= new BufferedReader( new InputStreamReader(url.openStream()));
                  String line;
                  while((line=br.readLine())!=null){
                      line=line.trim();
                      String pat="<img src=\"/(\\w*).jpg\" alt=\"Picture\"/></div>";
                       Pattern pattern = Pattern.compile(pat);
                       Matcher matcher = pattern.matcher(line);
                       boolean matchFound = matcher.find();
                        if (matchFound) {
                            String nombre=matcher.group(1);
                            String urlI=baseURL+"/file/"+nombre+".jpg";
                            Bajador baj= new Bajador(baseURL,nombre,par);
                            baj.start();
                          //  baja(baseURL,nombre);
                           
                        }
                  }
                } catch (MalformedURLException ex) {
                    Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
                } catch (IOException e) {
                    Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, e);
                }

Hope this help you.

That helped me quite a bit, I read into patterns and while it seems a bit confusing I understood enough to where I believe you are correct in saying I only needed to search from a-z and 0-9.

I am left a bit confused about some of your code, so if you don't mind I'll just ask the questions here for a better understanding.

if (matchFound) {
                            String nombre=matcher.group(1);
                            String urlI=baseURL+"/file/"+nombre+".jpg";
                            Bajador baj= new Bajador(baseURL,nombre,par);
                            baj.start();
                          //  baja(baseURL,nombre);
                           
                        }

I understand that your code is for pulling images off a website (which I might be using later) and I also understand that if a match is found this is where I tell it what to do, but I'm a bit curious about your code here.

The java docs left me a bit confused about the matcher.group. Would you mind explaining the piece of code above?

Instead of bajador, I would put a class that will add the finding to an arraylist and write it to file, unless there is a better option
----

You could use HTML Parser, which a Java library used to parse HTML in either a linear or nested fashion. It is an open source tool and can be found on SourceForge.
you could also useThe Swing HTML Parser.

Thanks for pointing this out, this looks like it cuts out a lot of the work :] I'll check into this as well. However, one thing is I'm also looking to interact with the web page too such as clicking buttons, I'm not sure if this would allow that.

----

Another question I have is that will I be able to manipulate menus and what not with similar code. Meaning, I am looking to also do such things through a search feature on forums, so would I be able to click buttons and select things from drop down menus?

[santiagozky]

The concept of a group is quite simple. Look at my regexp:

="<img src=\"/(\\w*).jpg\" alt=\"Picture\"/></div>";

Note that \\w* in within parenthesis. This tells the pattern to 'store' what is in that portion of the regular expression (the filename in my case), then you can access the stored string with matcher.group(1); (0 is the entire expression and 1 is the first group defined in the expression).

if your regular expression look like (car([0-9]*)) the the first group is (car([0-9]*)) (car,car1,car11,car16, etc) and the secound group is ([0-9]*) (empty,1,3,7,etc).

This is because sometimes you need to find a pattern and use information from it. Groups allow you to do this.

[KirkPatrick]

So essentially my if(matchFound) would be much simpler, like so?

ArrayList info = new Arraylist();

                        if (matchFound) {
                            String name=matcher.group(1);
                            info.add(name); //just add the info from the string inside title
                                                       
                        }
                      //print out info in arraylist
                      //write info from arraylist to file

Thank you for going into detail about the groups, that is quite helpful. Just curious, but why do you include: alt=\"Picture\"/></div> ?

[santiagozky]

Thank you for going into detail about the groups, that is quite helpful. Just curious, but why do you include: alt=\"Picture\"/></div> ?

the page had a lot of pictures, I only wanted the ones with that attribute. The program was intended to work with an specific site.

[KirkPatrick]

the page had a lot of pictures, I only wanted the ones with that attribute

Alright, thanks bud. I will continue working on this side project of mine and I'll be sure to report back on the progress.

I will need to look into expressions a bit more as i don't fully understand them all, some seem pretty complicated.

Also, are you aware of how to interact with websites? Such as the buttons, drop menus, etc?

[santiagozky]

Alright, thanks bud. I will continue working on this side project of mine and I'll be sure to report back on the progress.

I will need to look into expressions a bit more as i don't fully understand them all, some seem pretty complicated.

Also, are you aware of how to interact with websites? Such as the buttons, drop menus, etc?

Im glad to help.

No, I have never tried to do anything like that. I suppose that you would need a complete embedded browser (with javascript most likely) in order to use javascript to simulate the events. But I'm just thinking at loud

[KirkPatrick]

Sorry, I must have missed this in my previous posts, but what did your baseUrl and par look like?

I would assume it is something like this:

baseUrl = "http://www.daniweb.com/forums/forum9"
par = "-desc-"(0-9)".html"

Or am I thinking of it wrong?

[KirkPatrick]

I was testing it out and am wondering why it only prints out 1 letter at a time?

Here is my example code:

String link = "http://www.daniweb.com/forums/forum9.html";
        ArrayList pageInfo = new ArrayList();
        
        try {
            URL url = new URL(link);
            BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream()));
            
            String line;

            while((line = br.readLine()) != null) {
                line = line.trim();
                String pat = "<div class=\"([a-zA-Z0-9])+\""; //pattern check between a-z and 0-9 -
                Pattern pattern = Pattern.compile(pat);
                Matcher matcher = pattern.matcher(line);
                
                boolean matchFound = matcher.find();
                if (matchFound) {
                    String name = matcher.group(1);
                    pageInfo.add(name);
                    System.out.println(name);
                    
                }
            }
            
            System.out.println(pageInfo);
            
        } catch (MalformedURLException ex) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, ex);
        } catch (IOException e) {
            Logger.getLogger(Main.class.getName()).log(Level.SEVERE, null, e);
        }
        
    }

and here is what the output is:

p
d
g
x
c
c
c
c
v
i
e
e
v
x
c
c
c
c
2
x
c
c
c
c
t
d
m
t
[p, d, g, x, c, c, c, c, v, i, e, e, v, x, c, c, c, c, 2, x, c, c, c, c, t, d, m, t]

I assume I would have to change my pattern?

[santiagozky]

Yes, the problem is the pattern:

([a-zA-Z0-9])+

[a-zA-Z0-9] means 1 character letter o number.
+ means 'at least 1 time'.
But you are saying "at least one time the group consisting of one character" and you should say "the group consisting of at least one character"

[KirkPatrick]

I apologize that I haven't had the chance to reply back with progress. I have managed to get it to do what I was intending.

My next step is being able to click buttons and links through my java program. I believe I need to read up on http post.

Thanks for all your help, its much appreciated