httpClinet循环抓网页时出现异常

Question

java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at org.apache.commons.httpclient.ContentLengthInputStream.read(ContentLengthInputStream.java:170)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)
at org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:690)
at com.easylotto.crawler.SimplenessACrawler.snatchAt(SimplenessACrawler.java:74)
at com.easylotto.crawler.ExecuteCrawler.doCrawler(ExecuteCrawler.java:49)
at com.easylotto.crawler.ExecuteCrawler.main(ExecuteCrawler.java:54)循环及抓取模式
CrawlerResult cr=null;
for(int i=14049;i<=721168;i++){
   etMethod=new GetMethod(tempUrl);
   statusCode = httpClient.executeMethod(getMethod);
   ......
   getMethod.releaseConnection();
}
谁能告诉我解决办法...............

Answer 1

首先应该打印一下,statusCode,看看是什么?200?还是400,还是302.然后检查代码.CrawlerResult cr=null; 
for(int i=14049;i <=721168;i++){ 
  etMethod=new GetMethod(tempUrl); 
  statusCode = httpClient.executeMethod(getMethod); 
  System.out.println(statusCode);
  ...... 
  getMethod.releaseConnection(); 
} 
另外提点小建议.getMethod.releaseConnection(); 放在finally块中比较合适
for(int i=14049;i <=721168;i++){ //这个我不太明白是什么意思?这种循环意义不大.也许你另有用意,但我看不懂.呵呵
另外,在httpClient执行请求前,加上如下代码比较合适:
etMethod.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler());它可以帮你自动恢复请求时的错误.

Answer 2

我只能说让你试试。因为这个异常，多半是由于链接太快造成链接复位。所以你在开始和结束的时候都等会，或许能好。
呵呵。我不确定。good luck!