Member Avatar for gretty

Hello

I am new to Java & I am trying to find ways(in built Java objects/ways) to parse HTML. Can you suggest some objects in the Java Standard Library?

I have extended the object HTMLEditorKit.ParserCallback but when parsing a web pages' source code it literally takes 2 minutes or more! But some of that is my internet right now is only getting 9kb download per second :P but nvm that.

Here is what I have done to the HTMLEditorKit.ParserCallback (overloaded the handleStartTag() & handleText() functions ):

public class SearchResultGatherer extends HTMLEditorKit.ParserCallback implements Runnable
{

	/// Class Variables:
	
	int index = 0;
	
	private Vector <SearchResult> searchResults;
	private SearchEngine          searchEngine;
	private View                  appView;
	private double                searchTime;
	private int                   searchQuantity;	
	
	/// Class Methods:
	
	public SearchResultGatherer( Vector <SearchResult> _searchResults, SearchEngine _searchEngine, View _appView )
	{
		searchResults     = _searchResults;
		searchEngine      = _searchEngine;
		appView           = _appView;
		searchTime        = -1;
		searchQuantity    = -1;

		Thread thisThread = new Thread( this );
		thisThread.start();
		// Maybe do
		// thisThread.invokeAndWait();
	}
	
         public void handleStartTag( HTML.Tag t, MutableAttributeSet a, int pos ) 
	{
		for ( int i=0; i<searchEngine.targetElementInfo.length; i++ )
		{
			
			if ( t.toString().equals( searchEngine.targetElementInfo[i][0] ) )
			{
				if ( a.toString().equals( searchEngine.targetElementInfo[i][1] ) )
				{
					searchEngine.targetElementIdentified = true;
					return;
				}
				else if (i == 1)
				{
					System.out.println( "Element = " + t.toString() );
					System.out.println( "id      = " + a.toString() );
					searchEngine.targetElementIdentified = true;
					return;
				}
			}
			
		}
    }
	
	
	public void handleText( char[] arg0, int arg1 )
	{
		if ( searchEngine.targetElementIdentified )
		{
			System.out.println( arg0 );
			// System.out.println( "String arg = " + arg0.toString() );
			// System.out.println( "Int arg    = " + arg1 );
			Object searchData[]                  = searchEngine.retrieveSearchData( arg0 );
			searchTime                           = Double.parseDouble( searchData[0].toString() );
			searchQuantity                       = Integer.parseInt  ( searchData[1].toString() ); 
			//searchEngine.targetElementIdentified = false;
		}
	}

.....
  • 4 Contributors
  • 4 Replies
  • 215 Views
  • 4 Years Discussion Span
  • Latest Post Latest Post by olupotd

Dani AI

Generated

As discovered, Java SE provides only minimal HTML parsing (the Swing HTML support is handy for tiny, well-formed snippets but is brittle and often slow on real pages). Real-world HTML is usually malformed, large, or full of irrelevant bits, so practical solutions fall into three patterns that trade robustness, memory and API convenience.

  1. Streaming, SAX-style tolerant parsing — use a parser that produces SAX events from arbitrary HTML so the standard org.xml.sax handlers can be used. This keeps memory use low and is best when only a few fields must be extracted.
  2. DOM-style forgiving parsers — choose a parser that builds a DOM from malformed HTML if random access or structural queries are needed; this is heavier but simplifies extraction.
  3. Pre-clean-to-XML then reuse standard XML tools — run an HTML cleaner to emit well-formed XML and then use built-in SAX/DOM APIs if strict XML tooling is required.

Performance and robustness checklist (compliments to points from and ): avoid loading a Swing Document for large pages; stream the response directly into the parser; set connection timeouts and use buffered I/O; prefer ArrayList over synchronized Vector unless thread-safety is required; and profile whether time is spent on network, parsing or DOM allocation. Example: 

URLConnection conn = new URL(url).openConnection();
conn.setConnectTimeout(5000);
conn.setReadTimeout(15000);
try (InputStream in = new BufferedInputStream(conn.getInputStream())) {
    // feed 'in' to a streaming/tolerant parser
}

Additional cautions: never rely on fragile regex for full parsing; always detect character encoding from Content-Type or the document meta; test against several real pages (nested tags, unexpected attributes, missing end-tags). This summary complements the thread replies from , and and should help decide whether a streaming SAX approach, a tolerant DOM builder, or a cleaner-then-XML workflow is the best fit.

Recommended Answers

All 4 Replies

Member Avatar for gretty

No suggestions of a java HMTL parser? :(

Member Avatar for Taywin

Hmm... Parsing HTML text is not easy. I am not sure what's the purpose of your Java class? I would assume that you are printing out tag and its value??? How would you print out a nested value in a tag then? What would you do with ill-formatted HTML text? Would you be able to handle customized tag as well?

You could, however, implement a HTML text parser yourself in Java, but you need to be clear about each situation. Using other's library is OK as long as you completely understand what it does.

Member Avatar for peter_budo

You better to check out some other projects because above code is just very small portion of big idea, google results here

Member Avatar for olupotd

Try jsoup, it's pretty owesome and cool. get it at jsoup.org

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.