有些网站提示需要开启 cookie 才能采集数据,并不需要登录,仅需开启cookie就行,我用ie浏览是很正常的,以下是我一个获取网站源代码的函数,我在一个循环里面不断输入新的url用以获取网站源码,遇到需要开启cookie的网站就不行,
请问需要如何修改呢? public string GetHtml(string url,string theCoding)
{
string html = "";
HttpWebRequest httpWebRequest=null;
HttpWebResponse httpWebResponse=null;
Stream responseStream = null;
StreamReader streamReader = null; CookieContainer myCookieContainer = new CookieContainer(); //新建一个CookieContainer来存放Cookie集合
httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url); // url = "http://localhost/TestLogin/Default.aspx"
httpWebRequest.CookieContainer = myCookieContainer;
httpWebRequest.KeepAlive = true; try
{
httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
responseStream = httpWebResponse.GetResponseStream(); streamReader = new StreamReader(responseStream, Encoding.GetEncoding(theCoding));
html = streamReader.ReadToEnd();
}
catch (WebException ex)
{
html = "timeout";
}
finally
{
if (streamReader!=null)
streamReader.Dispose(); if(responseStream!=null)
responseStream.Dispose(); if(httpWebResponse!=null)
httpWebResponse.Close();
if (httpWebRequest!=null)
httpWebRequest.Abort();
}
return html;
}
请问需要如何修改呢? public string GetHtml(string url,string theCoding)
{
string html = "";
HttpWebRequest httpWebRequest=null;
HttpWebResponse httpWebResponse=null;
Stream responseStream = null;
StreamReader streamReader = null; CookieContainer myCookieContainer = new CookieContainer(); //新建一个CookieContainer来存放Cookie集合
httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url); // url = "http://localhost/TestLogin/Default.aspx"
httpWebRequest.CookieContainer = myCookieContainer;
httpWebRequest.KeepAlive = true; try
{
httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse();
responseStream = httpWebResponse.GetResponseStream(); streamReader = new StreamReader(responseStream, Encoding.GetEncoding(theCoding));
html = streamReader.ReadToEnd();
}
catch (WebException ex)
{
html = "timeout";
}
finally
{
if (streamReader!=null)
streamReader.Dispose(); if(responseStream!=null)
responseStream.Dispose(); if(httpWebResponse!=null)
httpWebResponse.Close();
if (httpWebRequest!=null)
httpWebRequest.Abort();
}
return html;
}
把这句代码提出来,不要每次都实例化,一直用同一个myCookieContainer 就可以了。
我试过 用一个全局的 CookieContainer myCookieContainer ,也就是一直用同一个myCookieContainer,发现时采集第一篇文章可以,但之后就被浏览器提示需要开启cookie,请问什么情况?
//CookieContainer myCookieContainer = new CookieContainer(); //新建一个CookieContainer来存放Cookie集合
httpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url); // url = "http://localhost/TestLogin/Default.aspx"
httpWebRequest.CookieContainer = publicCookieContainer ;
httpWebRequest.KeepAlive = true;
试试把.net framework的版本改高点,
我写了一个登录webQQ的程序,相同的代码3.5也报这个错,4.0就OK
{
myCookieContainer.Add(httpWebResponse.Cookies);
}