相信接觸過網絡開發的人對HTTP、HttpWebRequest、Socket這些東西都不陌生吧。它們之間的一些介紹和關系我這裡都忽略了。開我們平時開發過程中也是很少有機會接觸大什麼大並發這個東東，一般，這個我們以一個簡單demo來講解吧。
主要的調用關系圖如下：
類的結構圖如下：
public string Url：http請求的url字符串，如http://www.baidu.com/
public byte[] PostData：Post請求中的數據
public WebHeaderCollection Headers：請求的頭部數據
public bool AllowAutoRedirect ：是否允許301、302自動跳轉，如果你想獲取請求返回的頭部信息，建議一般設置為false
public Dictionary<string, string> ExternalData ：請求過程中附加的數據（如數據記錄的ID），便於在成功或失敗回調函數中調用
public Action<HttpContextInfo> ActionCompleted ：請求成功後所調用的函數
public Action<HttpRequestException> ActionException：請求失敗所調用函數
public HttpRequestInfo Clone()：返回當前對象的一個副本。
：
public Stream ResponseContent ：Http請求返回內容（除頭部信息）的對象流
public HttpStatusCode StatusCode：Http返回的狀態
public string StatusDescription ：Http狀態描述
public WebHeaderCollection Headers：Http返回的頭部信息
public string GetString(Encoding coding)：把http返回體中數據流轉換為字符串，轉換編碼就是我們所傳參數。
public interface
{
void GetResponseAsync(HttpRequestInfo request);
bool IsBusy { set; get; }
}
在IHttpRequest接口中，IsBusy屬性主要是判斷當前對象是否正在使用中，GetResponseAsync方法是真正完成Http請求的方法。
這裡我們主要看看HttpRequestFactory的封裝吧，管理對象實例的個數，相當於一個對象池，這裡的代碼主要是基於。net framework2.0的，
首先我們需要2個集合分別管理HttpRequestInfo請求實例和IHttpRequest處理請求實例，
static Queue<HttpRequestInfo> requestTask = new Queue<HttpRequestInfo>();
static List<IHttpRequest> Handlers = new List<IHttpRequest>();
而我們暴露給外部的一個主要方法就是AddRequestTask，
public static void AddRequestTask(HttpRequestInfo info)
{
if (!string.IsNullOrEmpty(info.Url))
{
lock (Lockobj)
{
}
}
}
那麼這些請求在什麼時候被處理了，在一個叫Process方法中處理，
Process( ( (task != && handler !=
在這個方法中我們需要調用來獲取處理對象實例，調用來獲取請求實例。如果這2個實例都存在我們調用 .GetResponseAsync();方法開始處理http請求。
GetAvailableHttpRequest如下：
( i = ; i < Handlers.Count; i++ (! (Handlers.Count <==
}
在方法中，我們首先在處理對象集合中查找是否有空閒對象，如果有就返回，否則檢查當前對象實例個數個數是否達到最大個數，如果沒有我們則創建新實例，且加入到集合中，再返回，否者返回null。所以在Process方法中有一個檢查，看啊看你返回的IHttpRequest是否為null，。GetTask和GetAvailableHttpRequest處理一樣。
那麼這裡的Process方法有在什麼地方調用了，在HttpRequestFactory的靜態構造函數中調用。
static HttpRequestFactory()
{
MaxRequestCount = 10;
}
到這裡我們的一個對象池就構造玩了。
二 現在我們來看看RequestHttpWebRequest是如何處理HTTP請求的。它主要使用HttpWebRequest來處理請求。
這裡我們主要使用HttpWebRequest的異步方法，因此我們需要構造一個狀態對象StateObjectInfo
class StateObjectInfo : HttpContextInfo
{
internal byte[] Buffer { set; get; } //把返回的流寫到.ResponseContent 時用到的暫存數組
internal Stream ReadStream { set; get; }//把返回的流寫到.ResponseContent
internal HttpWebRequest HttpWebRequest { set; get; }
internal RequestHttpWebRequest RequestHandler { set; get; }//主要便於後面改IsBusy屬性。
}
其GetResponseAsync實現如下：
webRequest,
}
其中的實現如下：
InitWebRequest(HttpRequestInfo info, HttpWebRequest webRequest, =
webRequest = HttpWebRequest.CreateDefault( Uri(info.Url))
= == = (info.Headers != && info.Headers.Count > ( key
state = = [ * == ==
關於該類的EndRequest、EndResponse我想就沒什麼說的了，其中ReadCallBack的實現如下：
= ar.AsyncState read = (read >
state.Buffer = =
這裡還有一個HandException方法需要我們注意：
}
這裡我們。
在我們來看看一個簡單的調用把：
DownLoadFile( remoteurl, destinationFilePath, (HasIllegalCharacters(destinationFilePath, , = = (! = Action<HttpContextInfo>(x => (x.ResponseInfo.StatusCode == (Stream wr = File.Open(destinationFilePath, FileMode.OpenOrCreate, FileAccess.Write), sr =[] data = [ * readcount = sr.Read(data, (readcount > = sr.Read(data, , message = DateTime.Now.ToString() + + remoteurl + +(ex =>= Regex( message ==== =
}
message = DateTime.Now.ToString() + + ex.HttpContextInfo.RequestInfo.Url + +
}
message = DateTime.Now.ToString() + + remoteurl + + HasIllegalCharacters( path, ( i = ; i < path.Length; i++ num2 = (((num2 == ) || (num2 == )) || (((num2 == ) || (num2 == )) || (num2 < (checkAdditional && ((num2 == ) || (num2 ==
對於這個調用的demo我這裡就不多說，
。
。
三 現在我們來看看SocketHttpRequest是如何處理HTTP請求的。它主要使用Socket來處理請求。
SocketHttpRequest和RequestHttpWebRequest一樣都是采用對象的異步模式，那麼也需要一個狀態對象：
class RequestSockeStateObject : HttpContextInfo
{
internal SocketHttpRequest RequestHandler { set; get; }
internal Socket _socket { set; get; } //普通http請求采用socket
internal List<byte> HeaderBuffer { set; get; }
internal byte[] Buffer { set; get; }
internal int ContentLength { set; get; }//http需要接收的數據長度
internal int ReceiveLength { set; get; }//http已經接收的數據長度
internal SslStream SslStream { set; get; }//https請求采用TcpClient，這裡需要用到SslStream
}
public void (HttpRequestInfo info)
{
RequestSockeStateObject _state;
InitRequestSockeStateObject(info, out _state);
(_state);
}
這裡的InitRequestSockeStateObject和RequestHttpWebRequest的InitWebRequest方法差不多，就不在累贅了。
主要看看方法：
_state._socket =
_state.SslStream =
_state._socket =
socket連接是需要IP和端口的，這裡我們借用來獲取所需端口，但是一台計算機的ip可能有很多個，實際只有一兩個可以連接，所以我們這裡需要調用方法，傳遞一個ip集合。如果是https的話，直接用socket我沒有搞定，最後用搞定，不知道大家有沒有其他方法。
其中Begin_Write、End_Write、Complete_Read方法是異步中所必需的方法，BeginSend、Send_Completed、Receive_Completed、RepeatReceive是socket異步中所需方法。其中Complete_Read和Receive_Completed方法相似。
= ar.AsyncState byteCount =
{
((state.ReceiveLength == state.ContentLength && state.ContentLength >
{
state.ResponseInfo.ResponseContent.Write(state.Buffer,
}
}
}
}
。這裡的方法如下：
}
}
state.RequestHandler.IsBusy =
EndReceive方法主要是關閉so
在SocketHttpRequest這個類中，我們是如何來獲取發送的http請求信息以及如何解析http返回的header信息了？
首先來看一個GetRequestData方法，它主要是通過RequestInfo實例來獲取請求信息：
namespace HttpRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
[Serializable]
public class HttpRequestInfo
{
public HttpRequestInfo(string url)
{
Url = url;
Headers = new WebHeaderCollection();
ExternalData = new Dictionary<string, string>();
}
public string Url { set; get; }
public byte[] PostData { set; get; }
public WebHeaderCollection Headers { set; get; }
public bool AllowAutoRedirect { set; get; }
public Dictionary<string, string> ExternalData { set; get; }
public Action<HttpContextInfo> ActionCompleted { set; get; }
public Action<HttpRequestException> ActionException { set; get; }
public HttpRequestInfo Clone()
{
HttpRequestInfo newobj = new HttpRequestInfo(this.Url)
{
AllowAutoRedirect = this.AllowAutoRedirect,
ActionCompleted = this.ActionCompleted,
ActionException = this.ActionException
};
if (PostData != null && PostData.Length > 0)
{
newobj.PostData = new byte[this.PostData.Length];
Array.Copy(this.PostData, 0, newobj.PostData, 0, this.PostData.Length);
}
if (Headers.Count > 0)
{
foreach (string key in Headers.Keys)
{
newobj.Headers.Add(key, Headers[key]);
}
}
if (ExternalData.Count > 0)
{
foreach (string key in ExternalData.Keys)
{
newobj.ExternalData.Add(key, ExternalData[key]);
}
}
return newobj;
}
}
public class HttpRequestException : Exception
{
public HttpRequestException(HttpContextInfo context, Exception ex)
: base(ex.Message, ex)
{
HttpContextInfo = context;
}
public HttpContextInfo HttpContextInfo { set; get; }
}
}
namespace HttpRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Net;
[Serializable]
public class HttpResponseInfo
{
public HttpResponseInfo()
{
ResponseContent = new MemoryStream();
Headers = new WebHeaderCollection();
}
public Stream ResponseContent { set; get; }
HttpStatusCode _statusCode;
public HttpStatusCode StatusCode
{
set
{
_statusCode = value;
StatusDescription = System.Web.HttpWorkerRequest.GetStatusDescription((int)_statusCode);
}
get
{
return _statusCode;
}
}
public string StatusDescription { set; get; }
public WebHeaderCollection Headers { set; get; }
public string GetString(Encoding coding)
{
StringBuilder str = new StringBuilder();
Stream sr = ResponseContent;
byte[] data = new byte[1024 * 100];
int readcount = sr.Read(data, 0, data.Length);
while (readcount > 0)
{
str.Append(coding.GetString(data, 0, readcount));
readcount = sr.Read(data, 0, data.Length);
}
ResponseContent.Seek(0, SeekOrigin.Begin);
return str.ToString();
}
}
public class HttpContextInfo
{
public HttpResponseInfo ResponseInfo { set; get; }
public HttpRequestInfo RequestInfo { set; get; }
}
public interface IHttpRequest
{
void GetResponseAsync(HttpRequestInfo request);
bool IsBusy { set; get; }
}
}
namespace HttpRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Data;
using System.Data.OleDb;
using System.Net;
using System.Xml;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
public class RequestHttpWebRequest : IHttpRequest
{
class StateObjectInfo : HttpContextInfo
{
internal byte[] Buffer { set; get; }
internal Stream ReadStream { set; get; }
internal HttpWebRequest HttpWebRequest { set; get; }
internal RequestHttpWebRequest RequestHandler { set; get; }
}
private bool CheckValidationResult(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors errors)
{
//直接確認，否則打不開
return true;
}
public void GetResponseAsync(HttpRequestInfo info)
{
HttpWebRequest webRequest;
StateObjectInfo state;
InitWebRequest(info, out webRequest, out state);
try
{
if (IsHttpPost)
{
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.BeginGetRequestStream(EndRequest, state);
}
else
{
webRequest.BeginGetResponse(EndResponse, state);
}
}
catch (Exception ex)
{
HandException(ex, state);
}
}
void EndRequest(IAsyncResult ar)
{
StateObjectInfo state = ar.AsyncState as StateObjectInfo;
try
{
HttpWebRequest webRequest = state.HttpWebRequest as HttpWebRequest;
using (Stream stream = webRequest.EndGetRequestStream(ar))
{
byte[] data = state.RequestInfo.PostData;
stream.Write(data, 0, data.Length);
}
webRequest.BeginGetResponse(EndResponse, state);
}
catch (Exception ex)
{
HandException(ex, state);
}
}
void EndResponse(IAsyncResult ar)
{
StateObjectInfo state = ar.AsyncState as StateObjectInfo;
try
{
HttpWebResponse webResponse = state.HttpWebRequest.EndGetResponse(ar) as HttpWebResponse;
state.ResponseInfo.StatusCode = webResponse.StatusCode;
state.ResponseInfo.StatusDescription = webResponse.StatusDescription;
foreach (string key in webResponse.Headers.AllKeys)
{
state.ResponseInfo.Headers.Add(key, webResponse.Headers[key]);
}
state.ReadStream = webResponse.GetResponseStream();
state.ReadStream.BeginRead(state.Buffer, 0, state.Buffer.Length, ReadCallBack, state);
}
catch (Exception ex)
{
HandException(ex, state);
}
}
void ReadCallBack(IAsyncResult ar)
{
StateObjectInfo state = ar.AsyncState as StateObjectInfo;
try
{
int read = state.ReadStream.EndRead(ar);
if (read > 0)
{
state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, read);
state.ReadStream.BeginRead(state.Buffer, 0, state.Buffer.Length, ReadCallBack, state);
}
else
{
state.ReadStream.Close();
state.HttpWebRequest.Abort();
if (state.RequestInfo.ActionCompleted != null)
{
state.ResponseInfo.ResponseContent.Seek(0, SeekOrigin.Begin);
state.RequestInfo.ActionCompleted(state);
}
state.Buffer = null;
state.RequestHandler.IsBusy = false;
}
}
catch (Exception ex)
{
HandException(ex, state);
}
}
private void InitWebRequest(HttpRequestInfo info, out HttpWebRequest webRequest, out StateObjectInfo state)
{
IsBusy = true;
if (info.PostData != null && info.PostData.Length > 0)
{
IsHttpPost = true;
}
else
{
IsHttpPost = false;
}
if (info.Url.ToLower().Trim().StartsWith("https"))
{
IsHttps = true;
ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback(CheckValidationResult);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
}
else
{
IsHttps = false;
}
webRequest = HttpWebRequest.CreateDefault(new Uri(info.Url)) as HttpWebRequest;
if (IsHttps)
{
/*基礎連接已經關閉: 發送時發生錯誤 */
/*無法從傳輸連接中讀取數據: 遠程主機強迫關閉了一個現有的連接*/
webRequest.KeepAlive = false;
webRequest.ProtocolVersion = HttpVersion.Version10;
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)";
}
webRequest.AllowAutoRedirect = info.AllowAutoRedirect;
if (info.Headers != null && info.Headers.Count > 0)
{
foreach (string key in info.Headers.Keys)
{
webRequest.Headers.Add(key, info.Headers[key]);
}
}
//webRequest.Proxy = WebProxy.GetDefaultProxy();
//webRequest.Proxy.Credentials = CredentialCache.DefaultCredentials;
//webResponse.Headers.Get("Set-Cookie");
state = new StateObjectInfo
{
Buffer = new byte[1024 * 100],
HttpWebRequest = webRequest,
RequestHandler = this,
RequestInfo = info,
ResponseInfo = new HttpResponseInfo()
};
}
private bool IsHttpPost { set; get; }
private bool IsHttps { set; get; }
public bool IsBusy { set; get; }
private void HandException(Exception ex, StateObjectInfo state)
{
if (state.ReadStream != null)
state.ReadStream.Close();
if (state.HttpWebRequest != null)
state.HttpWebRequest.Abort();
state.Buffer = null;
if (state.RequestInfo.ActionException != null)
{
state.RequestInfo.ActionException(new HttpRequestException(state, ex));
}
state.RequestHandler.IsBusy = false;
}
}
}
namespace HttpRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Net;
public class HttpRequestFactory
{
static HttpRequestFactory()
{
MaxRequestCount = 10;
ThreadPool.QueueUserWorkItem(new WaitCallback(Process));
}
static readonly object Lockobj = new object();
static long requestCount = 0;
static Queue<HttpRequestInfo> requestTask = new Queue<HttpRequestInfo>();
static List<IHttpRequest> Handlers = new List<IHttpRequest>();
public static void AddRequestTask(HttpRequestInfo info)
{
if (!string.IsNullOrEmpty(info.Url))
{
lock (Lockobj)
{
requestTask.Enqueue(info);
Interlocked.Increment(ref requestCount);
}
}
}
private static IHttpRequest GetAvailableHttpRequest()
{
lock (Lockobj)
{
for (int i = 0; i < Handlers.Count; i++)
{
if (!Handlers[i].IsBusy)
{
return Handlers[i];
}
}
if (Handlers.Count <= MaxRequestCount)
{
IHttpRequest handler = (IHttpRequest)Activator.CreateInstance(_httpRequestType);
Handlers.Add(handler);
return handler;
}
}
return null;
//return GetAvailableHttpRequest();
}
private static HttpRequestInfo GetTask()
{
HttpRequestInfo task = null;
lock (Lockobj)
{
if (requestTask.Count > 0)
{
task = requestTask.Dequeue();
return task;
}
}
return task;
}
private static void Process(object obj)
{
while (true)
{
IHttpRequest handler = GetAvailableHttpRequest();
while (handler == null)
{
Thread.Sleep(500);
handler = GetAvailableHttpRequest();
}
HttpRequestInfo task = GetTask();
while (task == null)
{
Thread.Sleep(500);
task = GetTask();
}
if (task != null && handler != null)
{
handler.GetResponseAsync(task);
Interlocked.Decrement(ref requestCount);
Thread.Sleep(50);
}
}
}
public static long TaskCount
{
get
{
return Interlocked.Read(ref requestCount);
}
}
static int _maxRequestCount = 2;
public static int MaxRequestCount
{
set
{
_maxRequestCount = value;
ServicePointManager.DefaultConnectionLimit = _maxRequestCount * 2;
}
get
{
return _maxRequestCount;
}
}
static Type _httpRequestType = typeof(RequestHttpWebRequest);
public static void SetHttpRequestType(Type type)
{
if (type.IsClass && typeof(IHttpRequest).IsAssignableFrom(type))
{
_httpRequestType = type;
}
}
}
}
namespace HttpRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading;
using System.Reflection;
using System.IO;
public class LogManager
{
static object lockobj = new object();
static Queue<string> Messages = new Queue<string>();
static long messageCount = 0;
static LogManager()
{
ThreadPool.QueueUserWorkItem(new WaitCallback(Work));
}
public static void LogException(Exception ex)
{
Type type = ex.GetType();
StringBuilder sb = new StringBuilder();
sb.AppendLine(type.ToString() + "--------" + DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss"));
PropertyInfo[] properties = type.GetProperties();
foreach (PropertyInfo p in properties)
{
if (p.PropertyType == typeof(string))
{
object msg = p.GetValue(ex, null);
if (msg != null)
sb.AppendLine(p.Name + ":" + msg.ToString());
}
}
lock (lockobj)
{
LogException(sb.ToString());
}
}
public static void LogException(string msg)
{
lock (lockobj)
{
Messages.Enqueue(msg);
Interlocked.Increment(ref messageCount);
}
}
static void Work(object obj)
{
if (!File.Exists(ExceptionLogFilePath))
{
FileStream fs = File.Create(ExceptionLogFilePath);
fs.Close();
}
while (true)
{
if (MessageCount > 0)
{
string msg = string.Empty;
lock (lockobj)
{
msg = Messages.Dequeue();
Interlocked.Decrement(ref messageCount);
if (!string.IsNullOrEmpty(msg))
{
using (StreamWriter sw = new StreamWriter(ExceptionLogFilePath, true, Encoding.UTF8))
{
sw.Write(msg);
sw.WriteLine();
sw.Flush();
}
}
}
}//end if
Thread.Sleep(500);
}
}//end
static long MessageCount
{
get
{
return Interlocked.Read(ref messageCount);
}
}
public static string ExceptionLogFilePath { set; get; }
}
}
SocketHttpRequest代碼：
namespace HttpRequest
{
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Text.RegularExpressions;
public class SocketHttpRequest : IHttpRequest
{
class RequestSockeStateObject : HttpContextInfo
{
internal SocketHttpRequest RequestHandler { set; get; }
internal Socket _socket { set; get; }
internal List<byte> HeaderBuffer { set; get; }
internal byte[] Buffer { set; get; }
internal int ContentLength { set; get; }
internal int ReceiveLength { set; get; }
internal SslStream SslStream { set; get; }
}
public void GetResponseAsync(HttpRequestInfo info)
{
RequestSockeStateObject _state;
InitRequestSockeStateObject(info, out _state);
SocketConnection(_state);
}
void SocketConnection(RequestSockeStateObject _state)
{
try
{
Uri uri = new Uri(_state.RequestInfo.Url);
IPHostEntry hostEntry = Dns.GetHostEntry(uri.Host);
if (IsHttps)
{
TcpClient tcpclient = new TcpClient();
tcpclient.Connect(hostEntry.AddressList, uri.Port);
_state._socket = tcpclient.Client;
SslStream sslStream = new SslStream(tcpclient.GetStream(), false, new RemoteCertificateValidationCallback(ValidateServerCertificate), null);
sslStream.AuthenticateAsClient(hostEntry.HostName, new X509CertificateCollection(), SslProtocols.Ssl3 | SslProtocols.Tls, false);
_state.SslStream = sslStream;
Begin_Write(_state);
}
else
{
Socket client = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
client.Connect(hostEntry.AddressList, uri.Port);
_state._socket = client;
BeginSend(_state);
}
}
catch (Exception ex)
{
HandException(ex, _state);
}
}
#region SSL Https
void Begin_Write(RequestSockeStateObject state)
{
try
{
byte[] requestData = GetRequestData(state);
state.SslStream.BeginWrite(requestData, 0, requestData.Length, End_Write, state);
}
catch (Exception ex)
{
HandException(ex, state);
}
}
protected virtual void End_Write(IAsyncResult ar)
{
RequestSockeStateObject state = ar.AsyncState as RequestSockeStateObject;
try
{
state.SslStream.EndWrite(ar);
state.SslStream.Flush();
state.SslStream.BeginRead(state.Buffer, 0, state.Buffer.Length, Complete_Read, state);
}
catch (Exception ex)
{
HandException(ex, state);
}
}
protected virtual void Complete_Read(IAsyncResult ar)
{
RequestSockeStateObject state = ar.AsyncState as RequestSockeStateObject;
try
{
int byteCount = state.SslStream.EndRead(ar);
if (state.ResponseInfo.Headers.Count < 1)
{
SetResponseHeaders(state, byteCount);
if ((state.ReceiveLength == state.ContentLength && state.ContentLength > 0))
{
EndReceive(state);
}
else
{
state.SslStream.BeginRead(state.Buffer, 0, state.Buffer.Length, Complete_Read, state);
}
}
else
{
if (byteCount > 0 && byteCount == state.Buffer.Length)
{
state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, byteCount);
state.SslStream.BeginRead(state.Buffer, 0, state.Buffer.Length, Complete_Read, state);
}
else
{
state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, byteCount);
EndReceive(state);
}
}
}
catch (Exception ex)
{
HandException(ex, state);
}
}
#endregion
#region Http
void BeginSend(RequestSockeStateObject state)
{
try
{
byte[] byteData = GetRequestData(state);
SocketAsyncEventArgs arg = new SocketAsyncEventArgs
{
UserToken = state,
RemoteEndPoint = state._socket.RemoteEndPoint
};
arg.SetBuffer(byteData, 0, byteData.Length);
arg.Completed += new EventHandler<SocketAsyncEventArgs>(Send_Completed);
bool send = state._socket.SendAsync(arg);
if (!send)
{
HandException(new SocketException((int)SocketError.NoData), state);
}
}
catch (Exception ex)
{
HandException(ex, state);
}
}
protected virtual void Send_Completed(object sender, SocketAsyncEventArgs e)
{
RequestSockeStateObject state = e.UserToken as RequestSockeStateObject;
try
{
if (e.SocketError == SocketError.Success)
{
SocketAsyncEventArgs arg = new SocketAsyncEventArgs
{
UserToken = state,
RemoteEndPoint = state._socket.RemoteEndPoint,
};
arg.Completed += new EventHandler<SocketAsyncEventArgs>(Receive_Completed);
arg.SetBuffer(state.Buffer, 0, state.Buffer.Length);
bool receive = state._socket.ReceiveAsync(arg);
if (!receive)
{
HandException(new SocketException((int)SocketError.NoData), state);
}
}
}
catch (Exception ex)
{
HandException(ex, state);
}
}
protected virtual void Receive_Completed(object sender, SocketAsyncEventArgs e)
{
RequestSockeStateObject state = e.UserToken as RequestSockeStateObject;
if (e.SocketError == SocketError.Success)
{
if (state.ResponseInfo.Headers.Count < 1)
{
SetResponseHeaders(state, e.BytesTransferred);
if ((state.ReceiveLength == state.ContentLength && state.ContentLength > 0))
{
EndReceive(state);
}
else
{
RepeatReceive(state);
}
}
else
{
if (e.BytesTransferred > 0 && e.BytesTransferred == state.Buffer.Length)
{
state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, e.BytesTransferred);
RepeatReceive(state);
}
else
{
state.ResponseInfo.ResponseContent.Write(state.Buffer, 0, e.BytesTransferred);
EndReceive(state);
}
}
}
else
{
HandException(new SocketException((int)e.SocketError), state);
}
}
//end
void RepeatReceive(RequestSockeStateObject state)
{
SocketAsyncEventArgs arg = new SocketAsyncEventArgs
{
UserToken = state,
RemoteEndPoint = state._socket.RemoteEndPoint,
};
arg.Completed += new EventHandler<SocketAsyncEventArgs>(Receive_Completed);
arg.SetBuffer(state.Buffer, 0, state.Buffer.Length);
bool receive = state._socket.ReceiveAsync(arg);
if (!receive)
{
HandException(new SocketException((int)SocketError.SocketError), state);
}
}
#endregion
void EndReceive(RequestSockeStateObject state)
{
/*
* if (state.RequestInfo.AllowAutoRedirect &&
(state.ResponseInfo.StatusCode == HttpStatusCode.Found || state.ResponseInfo.StatusCode == HttpStatusCode.MovedPermanently))
{
string location = state.ResponseInfo.Headers["Location"];
state.RequestInfo.Url = location;
state.RequestInfo.Headers = state.ResponseInfo.Headers;
state.RequestInfo.Headers.Remove("Location");
state.RequestInfo.Headers.Add("Referer", location);
Begin_Write(state);
}
*/
if (IsHttps)
{
state.SslStream.Close();
state.SslStream = null;
}
else
{
state._socket.Shutdown(SocketShutdown.Both);
state._socket.Close();
state._socket = null;
}
if (state.RequestInfo.ActionCompleted != null)
{
state.ResponseInfo.ResponseContent.Seek(0, SeekOrigin.Begin);
state.RequestInfo.ActionCompleted(state);
}
state.RequestHandler.IsBusy = false;
}
#region Set Response Header
/// <summary>
/// 判斷讀取的信息中是否包含返回頭的結束符，如果有則取出返回頭的信息
/// </summary>
/// <param name="state">狀態信息</param>
/// <param name="bytesRead">讀取的字節數</param>
void SetResponseHeaders(RequestSockeStateObject state, int bytesRead)
{
try
{
byte[] tempArray = new byte[bytesRead];
Array.Copy(state.Buffer, 0, tempArray, 0, bytesRead);
state.HeaderBuffer.AddRange(tempArray);
tempArray = state.HeaderBuffer.ToArray();
string headerSpilt = "\r\n\r\n";
byte[] headerbyte = Encoding.ASCII.GetBytes(headerSpilt);
int contentindex = DestTag(tempArray, headerbyte, 0, tempArray.Length);
if (contentindex > 0)
{
string headerStr = Encoding.ASCII.GetString(tempArray, 0, contentindex);
int startIndex = contentindex + headerbyte.Length;
SetResponseHeaders(headerStr, state);
state.ReceiveLength = tempArray.Length - startIndex;
state.ResponseInfo.ResponseContent.Write(tempArray, startIndex, tempArray.Length - startIndex);
state.HeaderBuffer.Clear();
}
}
catch (Exception ex)
{
HandException(ex, state);
}
}
/// <summary>
/// 設置ResponseInfo的返回頭，並且檢查返回頭是否含有Keep-Alive和Content-Length
/// </summary>
/// <param name="headerStr">字符串形式的返回頭</param>
/// <param name="state">整個異步處理過程中的對象</param>
void SetResponseHeaders(string headerStr, RequestSockeStateObject state)
{
try
{
string[] headers = headerStr.Split(new string[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries);
string statline = headers[0];
state.ResponseInfo.StatusCode = GetStatusCode(statline);
for (int i = 1; i < headers.Length; i++)
{
int index = headers[i].IndexOf(":");
if (index > 1)
{
string key = headers[i].Substring(0, index);
string value = headers[i].Substring(index + 1);
state.ResponseInfo.Headers.Add(key.Trim(), value.Trim());
}
}
string contentLength = state.ResponseInfo.Headers["Content-Length"];
state.ContentLength = int.Parse(contentLength);
state.ReceiveLength = 0;
}
catch (Exception ex)
{
HandException(ex, state);
}
}
HttpStatusCode GetStatusCode(string text)
{
Regex reg = new Regex(@"\d{3}");
Match m = reg.Match(text);
string code = m.Value;
HttpStatusCode statu = (HttpStatusCode)Enum.Parse(typeof(HttpStatusCode), code, true);
return statu;
}
/// <summary>
/// 在指定的數組中檢索短數組
/// </summary>
/// <param name="buffer">要檢索的大數組</param>
/// <param name="tag">待檢索的小字數組</param>
/// <param name="startIndex">檢完索的啟始索引</param>
/// <param name="maxLength">大數組的有效長度</param>
/// <returns>如檢索到，則返回啟始索引，否則返回-1</returns>
protected int DestTag(byte[] buffer, byte[] tag, int startIndex, int maxLength)
{
bool temp = true;
int endIndex = maxLength - tag.Length;
while (startIndex < maxLength - tag.Length)
{
temp = true;
int pos = Array.IndexOf(buffer, tag[0], startIndex, endIndex - startIndex);
if (pos < 0)
{
return -1;
}
for (int j = 0; j < tag.Length; j++) //匹配所有字節
{
if (buffer[pos + j] != tag[j])
{
if (pos > startIndex)
startIndex = pos;
else
startIndex++;
temp = false;
break;
}
}
if (temp == true)
{
return pos;
}
}
return -1;
}
#endregion
/// <summary>
/// 根據指定的uri獲取請求頭
/// </summary>
/// <param name="uri">請求的uri</param>
/// <returns>以字符串的形式返回請求頭</returns>
byte[] GetRequestData(RequestSockeStateObject _state)
{
StringBuilder bufRequest = new StringBuilder();
Uri uri = new Uri(_state.RequestInfo.Url);
if (!IsHttpPost)
{
bufRequest.Append("GET ").Append(uri.OriginalString).AppendLine(" HTTP/1.1");
}
else
{
bufRequest.Append("POST ").Append(uri.OriginalString).AppendLine(" HTTP/1.1");
string contentLengthkey = "Content-Length";
string contentTypekey = "Content-Type";
List<string> headerKeys = new List<string>(_state.RequestInfo.Headers.AllKeys);
if (headerKeys.Contains(contentLengthkey))
{
_state.RequestInfo.Headers.Remove(contentLengthkey);
}
if (headerKeys.Contains(contentTypekey))
{
_state.RequestInfo.Headers.Remove(contentTypekey);
}
_state.RequestInfo.Headers.Add(contentTypekey, "application/x-www-form-urlencoded");
_state.RequestInfo.Headers.Add(contentLengthkey, _state.RequestInfo.PostData.Length.ToString());
}
_state.RequestInfo.Headers.Add("Host", uri.Host);
_state.RequestInfo.Headers.Add("Connection", "keep-alive");
if (_state.RequestInfo.Headers.Count > 0)
{
bufRequest.Append(_state.RequestInfo.Headers.ToString());
}
byte[] byteData = Encoding.ASCII.GetBytes(bufRequest.ToString());
if (!IsHttpPost)
{
return byteData;
}
else
{
byte[] sendData = new byte[byteData.Length + _state.RequestInfo.PostData.Length];
Array.Copy(byteData, 0, sendData, 0, byteData.Length);
Array.Copy(_state.RequestInfo.PostData, 0, sendData, byteData.Length, _state.RequestInfo.PostData.Length);
return sendData;
}
}
/// <summary>
/// 初始化State以及ResponseInfo
/// </summary>
/// <param name="info">請求對象</param>
void InitRequestSockeStateObject(HttpRequestInfo info, out RequestSockeStateObject _state)
{
this.IsBusy = true;
_state = new RequestSockeStateObject()
{
RequestHandler = this,
Buffer = new byte[1024 * 100],
HeaderBuffer = new List<byte>(),
RequestInfo = info,
ResponseInfo = new HttpResponseInfo()
};
if (info.PostData == null || info.PostData.Length < 1)
{
IsHttpPost = false;
}
else
{
IsHttpPost = true;
}
if (info.Url.ToLower().Trim().StartsWith("https"))
{
IsHttps = true;
}
else
{
IsHttps = false;
}
}
void HandException(Exception ex, RequestSockeStateObject state)
{
if (IsHttps)
{
if (state.SslStream != null)
state.SslStream.Close();
else if (state._socket != null)
{
state._socket.Shutdown(SocketShutdown.Both);
state._socket.Close();
}
}
else
{
state._socket.Shutdown(SocketShutdown.Both);
state._socket.Close();
}
if (state.RequestInfo.ActionException != null)
{
state.RequestInfo.ActionException(new HttpRequestException(state, ex));
}
state.RequestHandler.IsBusy = false;
}
static bool ValidateServerCertificate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
/*
if (sslPolicyErrors == SslPolicyErrors.None)
return true;
Console.WriteLine("Certificate error: {0}", sslPolicyErrors);
return false;
*/
return true;
}
bool IsHttps { set; get; }
bool IsHttpPost { set; get; }
public bool IsBusy
{
set;
get;
}
}
}
調用代碼：
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Data;
using System.Data.OleDb;
using System.Data.SqlClient;
using System.Diagnostics;
using System.IO;
using System.Net;
using System.Reflection;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
namespace HttpRequest
{
class Program
{
static void Main(string[] args)
{
LogManager.ExceptionLogFilePath = ConfigurationManager.AppSettings["ExceptionLogFilePath"].Trim();
ConnectionString = ConfigurationManager.AppSettings["ConnectionString"].Trim();
string remoteHostUrl = ConfigurationManager.AppSettings["remoteHostUrl"].Trim();
string destinationBasePath = ConfigurationManager.AppSettings["destinationBasePath"].Trim();
HttpRequestFactory.MaxRequestCount = Convert.ToInt32(ConfigurationManager.AppSettings["MaxRequestCount"]);
try
{
DateTime startTime = DateTime.Now;
Console.WriteLine("Start Time:" + startTime.ToLongTimeString());
AsyncDownLoadFiles(remoteHostUrl, destinationBasePath, null, true);
DateTime endtime = DateTime.Now;
Console.WriteLine("End Time:" + endtime.ToLongTimeString());
TimeSpan tm = endtime - startTime;
Console.WriteLine(tm.Hours.ToString() + " Hours " + tm.Minutes.ToString() + " Minutes " + tm.Seconds.ToString() + " Seconds");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
Console.ReadLine();
}
public static void AsyncDownLoadFiles(string remoteHostUrl, string destinationBasePath, List<string> tables, bool download)
{
try
{
List<string> archiveTables = tables;
if (archiveTables == null || archiveTables.Count < 1)
{
archiveTables = GetArchiveTables();
}
foreach (string tablename in archiveTables)
{
string sql = "SELECT ID,file_name_path FROM dbo.Archive_Files WHERE TableName ='" + tablename + "' AND Copyed=0 ORDER BY ID ASC";
DataTable dt = GetData(sql);
foreach (DataRow row in dt.Rows)
{
string id = row["ID"].ToString();
string file_name_path = row["file_name_path"].ToString();
file_name_path = file_name_path.Substring(2);
if (download)
{
AsyncDownLoadFile(remoteHostUrl + file_name_path, destinationBasePath + file_name_path, id);
}
else
{
CheckFileExists(destinationBasePath + file_name_path, id);
}
}
while (download)
{
if (HttpRequestFactory.TaskCount < HttpRequestFactory.MaxRequestCount)
{
break;
}
Thread.Sleep(10000);
}
}//end foreach
while (download)
{
if (HttpRequestFactory.TaskCount < 1)
{
break;
}
Thread.Sleep(10000);
}
bool finishedDownLoad = HasFinishedDownLoad();
int times = 0;
while (!finishedDownLoad && times < HttpRequestFactory.TaskCount)
{
Thread.Sleep(10000);
finishedDownLoad = HasFinishedDownLoad();
times++;
}
}
catch (Exception ex)
{
string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.Message;
LogManager.LogException(message);
}
}
public static void AsyncDownLoadFile(string remoteurl, string destinationFilePath, string id)
{
try
{
if (HasIllegalCharacters(destinationFilePath, true))
{
SetFileCopyed(id, "400", "HasIllegalCharacters");
return;
}
DirectoryInfo dir = new DirectoryInfo(destinationFilePath);
FileInfo destinationFile = new FileInfo(destinationFilePath);
if (!destinationFile.Directory.Exists)
{
destinationFile.Directory.Create();
}
HttpRequestInfo request = new HttpRequestInfo(remoteurl);
request.ActionCompleted = new Action<HttpContextInfo>(x =>
{
try
{
if (x.ResponseInfo.StatusCode == HttpStatusCode.OK)
{
using (Stream wr = File.Open(destinationFilePath, FileMode.OpenOrCreate, FileAccess.Write), sr = x.ResponseInfo.ResponseContent)
{
byte[] data = new byte[1024 * 100];
int readcount = sr.Read(data, 0, data.Length);
while (readcount > 0)
{
wr.Write(data, 0, readcount);
readcount = sr.Read(data, 0, data.Length);
}
}
SetFileCopyed(id, "200", string.Empty);
}
else
{
SetFileCopyed(id, ((int)x.ResponseInfo.StatusCode).ToString(), x.ResponseInfo.StatusDescription);
}
}
catch (Exception ea)
{
SetFileCopyed(id, "-1", ea.Message);
}
});
request.ActionException = new Action<HttpRequestException>(ex =>
{
try
{
Regex reg = new Regex(@"\d{3}", RegexOptions.Compiled);
string message = ex.Message;
Match m = reg.Match(message);
if (m.Success)
{
SetFileCopyed(id, m.Value, message);
}
else
{
SetFileCopyed(id, "503", message);
HttpRequestInfo newRequest = ex.HttpContextInfo.RequestInfo.Clone();
request.ActionCompleted = null;
request.ActionException = null;
HttpRequestFactory.AddRequestTask(newRequest);
}
}
catch (Exception ea)
{
SetFileCopyed(id, "-1", ea.Message);
}
});
HttpRequestFactory.AddRequestTask(request);
}
catch (Exception ex)
{
SetFileCopyed(id, "-1", ex.Message);
}
}
private static void CheckFileExists(string destinationFilePath, string id)
{
try
{
if (HasIllegalCharacters(destinationFilePath, true))
{
SetFileCopyed(id, "400", "HasIllegalCharacters");
return;
}
FileInfo destinationFile = new FileInfo(destinationFilePath);
if (destinationFile.Exists)
{
SetFileCopyed(id, "200", string.Empty);
}
else
{
SetFileCopyed(id, "404", "Not Found");
}
}
catch (Exception ex)
{
string message = ex.Message;
SetFileCopyed(id, "-1", message);
//message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + destinationFilePath + " : " + message;
//LogManager.LogException(message);
}
}
internal static bool HasIllegalCharacters(string path, bool checkAdditional)
{
for (int i = 0; i < path.Length; i++)
{
int num2 = path[i];
if (((num2 == 0x22) || (num2 == 60)) || (((num2 == 0x3e) || (num2 == 0x7c)) || (num2 < 0x20)))//" ' > | space
{
return true;
}
if (checkAdditional && ((num2 == 0x3f) || (num2 == 0x2a)))//? *
{
return true;
}
}
return false;
}
private static List<string> GetArchiveTables()
{
string sql = "SELECT DISTINCT TableName FROM dbo.Archive_Files ORDER BY TableName";
DataTable dt = GetData(sql);
List<string> archiveTables = new List<string>();
foreach (DataRow row in dt.Rows)
{
archiveTables.Add(row["TableName"].ToString());
}
return archiveTables;
}
static bool HasFinishedDownLoad()
{
string sql = "SELECT COUNT(*) FROM dbo.Archive_Files WITH(NOLOCK) WHERE Copyed=0";
return ExecuteScalar(sql) == 0;
}
private static bool SetFileCopyed(string id, string statusCode, string error)
{
string sql = string.Format("UPDATE dbo.Archive_Files SET Copyed={0}, CopyDate=GETDATE() ", statusCode);
if (!string.IsNullOrEmpty(error))
{
sql += string.Format(" ,CopyError='{0}' ", error);
}
sql += string.Format(" WHERE ID={0}", id);
return ExecuteCmd(sql);
}
private static DataTable GetData(string sql)
{
DataTable dt = new DataTable();
try
{
using (SqlConnection con = new SqlConnection(ConnectionString))
{
SqlDataAdapter adapter = new SqlDataAdapter(sql, con);
adapter.Fill(dt);
}
}
catch (Exception ex)
{
string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.Message;
LogManager.LogException(message);
}
return dt;
}
private static bool ExecuteCmd(string sql)
{
int resultCount = 0;
try
{
using (SqlConnection connection = new SqlConnection(ConnectionString))
{
SqlCommand cmd = new SqlCommand(sql, connection);
connection.Open();
cmd.CommandTimeout = 2 * 60;
resultCount = cmd.ExecuteNonQuery();
connection.Close();
}
}
catch (Exception ex)
{
string message = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss") + " : " + ex.Message;
LogManager.LogException(message);
}
return resultCount > 0;
}
static int ExecuteScalar(string sql)
{
int result = 0;
using (SqlConnection con = new SqlConnection(ConnectionString))
{
SqlCommand cmd = new SqlCommand(sql, con);
cmd.CommandTimeout = 2 * 60;
con.Open();
result = Convert.ToInt32(cmd.ExecuteScalar());
con.Close();
}
return result;
}
public static string ConnectionString { set; get; }
}
}
有不對的地方還請大家拍磚！
