关于java.net包中的URLConnection类的一些问题

Question

这是HttpConnection 我想请求某一个页面,但是总是发现有的可以正常显示,有的不能正常显示,比如"http://www.csdn.net"
是可以的,但是换成其他网站"http://www.runoob.com/","http://www.yq1012.com/"又不行了,只能显示开头的一部分数据,还是乱码,也不知道是是网站限制了这样的请求,还是我的代码有问题,谁能帮看一下,应该怎样请求,才能正常读出以上两个有问题网站的内容啊.URL localURL = new URL("http://www.csdn.net");
URLConnection connection = localURL.openConnection();
HttpURLConnection httpURLConnection = (HttpURLConnection)connection;
httpURLConnection.setRequestMethod("GET");
httpURLConnection.setRequestProperty("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
httpURLConnection.setRequestProperty("Accept-Language","zh-CN,zh;q=0.8");
httpURLConnection.setRequestProperty("Connection","keep-alive");
InputStream in =httpURLConnection.getInputStream();
byte[] buf =new byte[4096];
int len = in.read(buf);
System.out.println(new String(buf,0,len));

Answer 1

public static void main(String[] args) throws IOException {
         getURLByHtmlGet("http://www.runoob.com/","",""); 
}

public static String getURLByHtmlGet(String http_url_pro,String Param,String header) throws IOException {
URL url = new URL(http_url_pro+"?"+Param);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
if(header!=null){
header=header.split("\\) ")[0];
}
conn.setRequestProperty("accept", "text/html");
        conn.setRequestProperty("connection", "utf-8");
        conn.setRequestProperty("content-type","text/html;charset=utf-8");
conn.setRequestProperty("User-Agent", header+")"); System.out.println(conn.getRequestProperties());
InputStream inputStream =null;
try {
inputStream = conn.getInputStream();   //通过输入流获得网站数据
} catch (Exception e) {
try {
conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("accept", "text/html");
        conn.setRequestProperty("connection", "utf-8");
        conn.setRequestProperty("content-type","text/html;charset=utf-8");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X)");
System.out.println(conn.getRequestProperties());
inputStream = conn.getInputStream();   //通过输入流获得网站数据
} catch (Exception e2) {
try {
conn = (HttpURLConnection)url.openConnection();
conn.setRequestProperty("accept", "text/html");
        conn.setRequestProperty("connection", "utf-8");
        conn.setRequestProperty("content-type","text/html;charset=utf-8");
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64)");
System.out.println(conn.getRequestProperties());
inputStream = conn.getInputStream();   //通过输入流获得网站数据
} catch (Exception e3) {
e3.printStackTrace();
}
}
}
 

byte[] getData = readInputStream(inputStream);     //获得网站的二进制数据

String data = new String(getData, "UTF-8");
System.out.println(data);
return data;

} public static  byte[] readInputStream(InputStream inputStream) throws IOException {
byte[] buffer = new byte[1024];
int len = 0;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
while((len = inputStream.read(buffer)) != -1) {
bos.write(buffer, 0, len);
}

bos.close();
return bos.toByteArray();
}