Member Avatar for yosi501r

Hi everybody,

I want to get the HTML source code of a page like:

(the page is in hebrew)

But, after entering a hebrew page the characters I get are like
� ׳�׳ ׳’׳�׳™ ׳¢׳‘׳¨׳™ ׳¢׳™׳‘׳¨׳™, ׳�׳™׳�׳•׳� ׳�׳ ׳’׳�׳™

I want to see the Hebrew as it is.

The current code:

public static void main(String[] args) throws Exception {
        Scanner input = new Scanner(System.in);
        String a= input.next();
        URL yahoo = new URL(a);
        URLConnection yc = yahoo.openConnection();

        BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));


        String inputLine;
        
        

    	JTextArea tt= new JTextArea();
    	JFrame f = new JFrame();
    	f.add(tt);
    	f.setVisible(true);
    	JScrollPane bar = new JScrollPane();
    	tt.setAutoscrolls(true);
    	tt.add(bar);
    	
        while ((inputLine = in.readLine()) != null) {
        	

        	tt.append("\n"+inputLine);

        }
  
    	a=input.next();
    	System.out.print(a);
        in.close();
    }
}

How to solve the issue.

Any help will be appreciated!

  • 2 Contributors
  • 5 Replies
  • 197 Views
  • 3 Days Discussion Span
  • Latest Post Latest Post by mKorbel

Dani AI

Generated

This is classic mojibake: the page bytes are being decoded with the wrong character encoding. Your code creates an InputStreamReader without specifying the page charset, so Java falls back to the JVM/OS default charset (not necessarily the page’s charset). As hinted, you must detect and use the page’s actual charset instead of relying on defaults. (docs.oracle.com)

Practical approach — prefer the HTTP header first. After opening the connection inspect the response Content-Type (e.g. "text/html; charset=UTF-8"); if a charset parameter is present, use it when constructing the reader or when decoding the bytes. Example (header parsing + reader):

HttpURLConnection conn = (HttpURLConnection) new URL(url).openConnection();
conn.connect();
String ctype = conn.getContentType(); // e.g. "text/html; charset=UTF-8"
String charset = null;
if (ctype != null) {
  for (String p : ctype.replace(" ", "").split(";")) {
    if (p.toLowerCase().startsWith("charset=")) { charset = p.split("=",2)[1]; break; }
  }
}
Reader r = (charset != null)
  ? new InputStreamReader(conn.getInputStream(), charset)
  : new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8);

Use the Content-Type header as the authoritative hint from the server. (docs.oracle.com)

If the header has no charset, examine the HTML head for a meta charset (must appear early in the document — e.g. within the first 1K bytes) or use a library that does this automatically. A robust pattern: read the response into a byte buffer, inspect the first few KB for a meta charset or http-equiv value, then decode the entire byte array with the detected charset; otherwise fall back to UTF‑8. Or let Jsoup handle detection for you — it already implements this logic. (developer.mozilla.org)

For Hebrew pages prefer UTF‑8, but some legacy sites still use Windows‑1255 (cp1255) rather than cp1250 (which is not Hebrew). Also ensure your Swing control can render Hebrew: pick a font that contains Hebrew glyphs and set right-to-left orientation (e.g. setComponentOrientation(ComponentOrientation.RIGHT_TO_LEFT)) before calling setText(...). (en.wikipedia.org)

Summary: read the Content-Type header, fall back to meta charset in the head, then decode bytes with the detected charset and display using a Hebrew-capable font — that will stop the gibberish.

Recommended Answers

All 5 Replies

Member Avatar for mKorbel

if you have WinOS, then you have a lots of problems with localizations and CharEncode, you needed only add (maybe always) corrext EncodePage (Charset) for File and Streams

String fileEncoding = System.getProperty("file.encoding");
System.out.println("File Encoding: " + fileEncoding);
System.out.println("Char Encoding: " + charEncoding);
System.out.println("Char Encoding: " + Charset.availableCharsets());

InputStream in = null;
in = conn.getInputStream();
int len;
byte[] buf = new byte[1024];
while ((len = in.read(buf)) > 0) {
  bos.write(buf, 0, len);
}
String charEncoding = Charset.defaultCharset().name();
charEncoding = "cp1250"; //Slovak EncodePage
ret = new String(bos.toByteArray(), charEncoding);
ret1 = bos.toString(charEncoding);
Edited by mKorbel because: n/a
Member Avatar for yosi501r

if you have WinOS, then you have a lots of problems with localizations and CharEncode, you needed only add (maybe always) corrext EncodePage (Charset) for File and Streams

String fileEncoding = System.getProperty("file.encoding");
System.out.println("File Encoding: " + fileEncoding);
System.out.println("Char Encoding: " + charEncoding);
System.out.println("Char Encoding: " + Charset.availableCharsets());

InputStream in = null;
in = conn.getInputStream();
int len;
byte[] buf = new byte[1024];
while ((len = in.read(buf)) > 0) {
  bos.write(buf, 0, len);
}
String charEncoding = Charset.defaultCharset().name();
charEncoding = "cp1250"; //Slovak EncodePage
ret = new String(bos.toByteArray(), charEncoding);
ret1 = bos.toString(charEncoding);

Thank you, but I don't know how this gets combined with my code. I tried it but with errors.

Member Avatar for mKorbel

I think that every (maybe my mistake) File and Stream definitions allows 2nd. parameters for CharEncode, that's for suck/put data from web, get/put htmlPage, load/save File contents ...., for/from GUI

http://download.oracle.com/javase/tutorial/essential/io/index.html

http://www.java2s.com/Code/Java/File-Input-Output/CatalogFile-Input-Output.htm

http://www.java2s.com/Code/Java/File-Input-Output/CatalogFile-Input-Output.htm

try google for CharsetDecoder too

Member Avatar for yosi501r

I think that every (maybe my mistake) File and Stream definitions allows 2nd. parameters for CharEncode, that's for suck/put data from web, get/put htmlPage, load/save File contents ...., for/from GUI

http://download.oracle.com/javase/tutorial/essential/io/index.html

http://www.java2s.com/Code/Java/File-Input-Output/CatalogFile-Input-Output.htm

http://www.java2s.com/Code/Java/File-Input-Output/CatalogFile-Input-Output.htm

try google for CharsetDecoder too

Still the same, for hebrew sites I get gibberish.

Member Avatar for mKorbel

don't (me) silly, be sure that I can found

System.out.println("Char Encoding: " + Charset.availableCharsets());

(google + h) and read some chars rightToLeft, works both ISO-xxxx-x and windows-xxxx

I'm outta from this thread

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.