Heritrix

图10-37 Job提交后的Console界面（3）在面版的右测，它显示了当前Java虚拟机的一些状态，如图10-38所示，可以看到当前的堆大小为4184KB，而已经被使用了3806KB，另外，最大的堆内容可以达到65088KB，也就是在64M左右。

图10-38 内存状态显示（4）此时，单击面版中的“Start”链接，就会将此时处于“Pending”状态的抓取任务激活，令其开始抓取（5）在图10-39中，刚才还处于“Start”状态的链接已经变为了Hold状态。

这表明，抓取任务已经被激活。

图10-39 抓取开始（6）此时，面版中出现了一条抓取状态栏，它清楚的显示了当前已经被抓取的链接数量，另外还有在队列中等待被抓取的链接数量，然后用一个百分比显示出来。

（7）在绿红相间的长条左侧，是几个实时的运行状态，其中包括抓取的平均速度（KB/s）和每秒钟抓取的链接数（URIs/sec），另外的统计还包括抓取任务所消耗的时间和剩余的时间，不过这种剩余时间一般都不准，因为URI的数量总是在不断变化，每当分析一个网页，就会有新的URI加入队列中。

如图10-40所示。

图10-40 抓取的速度和时间（8）在绿红相间的长条右侧，是当前的负载，它显示了当前活跃的线程数量，同时，还统计了Heritrix内部的所有队列的平均长度。

如图10-41所示。

图10-41 线程和队列负载（9）从图10-40和图10-41中看到，真正的抓取任务还没有开始，队列中的总URI数量，以及下载的速率都还基本为0。

这应该还处于接收种子URL的网页信息的阶段。

让我们再来看一下当Heritrix运行一段时间后，整个系统的资源消耗和进度情况。

（10）在图10-42中，清楚的看到系统的资源消耗。

其中，每秒下载的速率已经达到了23KB，另外，平均每秒有19.3个URI被抓取。

在负载方面，初设的50个线程均处于工作状态，最长的队列长度已经达到了415个URI，平均长度为5。

Heritrix3.1.0源码解析（⼆⼗⼆）本⽂继续分析Heritrix3.1.0系统的源码，其实本⼈感觉接下来待分析的问题不是⼀两篇⽂章能够澄清，本⼈不能因为迫于表述⽽乱了问题本⾝的章法，接下来的分析的Heritrix3.1.0系统封装HttpClient组件可能要分⼏篇⽂章来解析我们知道，Heritrix3.1.0系统是通过封装HttpClient组件（⾥⾯封装了Socket）来与服务器通信的，Socket的输出流写⼊数据，输⼊流接收数据那么Heritrix3.1.0系统是怎样封装Httpclient（Heritrix3.1.0系统是采⽤的以前的Apache版本）组件的呢?我们可以看到，在FetchHTTP处理器⾥⾯有⼀段静态代码块，⽤于注册Socket⼯⼚，分别⽤于HTTP通信与HTTPS通信协议（基于TCP协议通信，⾄于两者的关系本⽂就不再分析了，不懂的读者可以参考⽹络通信⽅⾯的教程）/*** 注册http和https协议*/static {Protocol.registerProtocol("http", new Protocol("http",new HeritrixProtocolSocketFactory(), 80));try {ProtocolSocketFactory psf = new HeritrixSSLProtocolSocketFactory();Protocol p = new Protocol("https", psf, 443);Protocol.registerProtocol("https", p);} catch (KeyManagementException e) {e.printStackTrace();} catch (KeyStoreException e) {e.printStackTrace();} catch (NoSuchAlgorithmException e) {e.printStackTrace();}}上⾯的两个类HeritrixProtocolSocketFactory和HeritrixSSLProtocolSocketFactory都实现了HttpClient组件的ProtocolSocketFactory接⼝，⽤于创建客户端Socket对象（HeritrixSSLProtocolSocketFactory类间接实现了ProtocolSocketFactory接⼝）ProtocolSocketFactory接⼝定义了创建SOCKET对象的⽅法（package mons.httpclient.protocol）/*** A factory for creating Sockets.** <p>Both {@link ng.Object#equals(ng.Object) Object.equals()} and* {@link ng.Object#hashCode() Object.hashCode()} should be overridden appropriately.* Protocol socket factories are used to uniquely identify <code>Protocol</code>s and* <code>HostConfiguration</code>s, and <code>equals()</code> and <code>hashCode()</code> are* required for the correct operation of some connection managers.</p>** @see Protocol** @author Michael Becke* @author <a href="mailto:mbowler@">Mike Bowler</a>** @since 2.0*/public interface ProtocolSocketFactory {/*** Gets a new socket connection to the given host.** @param host the host name/IP* @param port the port on the host* @param localAddress the local host name/IP to bind the socket to* @param localPort the port on the local machine** @return Socket a new socket** @throws IOException if an I/O error occurs while creating the socket* @throws UnknownHostException if the IP address of the host cannot be* determined*/Socket createSocket(String host,int port,InetAddress localAddress,int localPort) throws IOException, UnknownHostException;/*** Gets a new socket connection to the given host.** @param host the host name/IP* @param port the port on the host* @param localAddress the local host name/IP to bind the socket to* @param localPort the port on the local machine* @param params {@link HttpConnectionParams Http connection parameters}** @return Socket a new socket** @throws IOException if an I/O error occurs while creating the socket* @throws UnknownHostException if the IP address of the host cannot be* determined* @throws ConnectTimeoutException if socket cannot be connected within the* given time limit** @since 3.0*/Socket createSocket(String host,int port,InetAddress localAddress,int localPort,HttpConnectionParams params) throws IOException, UnknownHostException, ConnectTimeoutException;/*** Gets a new socket connection to the given host.** @param host the host name/IP* @param port the port on the host** @return Socket a new socket** @throws IOException if an I/O error occurs while creating the socket* @throws UnknownHostException if the IP address of the host cannot be* determined*/Socket createSocket(String host,int port) throws IOException, UnknownHostException;}HeritrixProtocolSocketFactory类实现了上⾯的ProtocolSocketFactory接⼝（⽤于HTTP通信）public class HeritrixProtocolSocketFactory implements ProtocolSocketFactory {/*** Constructor.*/public HeritrixProtocolSocketFactory() {super();}@Overridepublic Socket createSocket(String host, int port, InetAddress localAddress,int localPort) throws IOException, UnknownHostException {// TODO Auto-generated method stubreturn new Socket(host, port, localAddress, localPort);}@Overridepublic Socket createSocket(String host, int port, InetAddress localAddress,int localPort, HttpConnectionParams params) throws IOException,UnknownHostException, ConnectTimeoutException {// TODO Auto-generated method stub// Below code is from the DefaultSSLProtocolSocketFactory#createSocket// method only it has workarounds to deal with pre-1.4 JVMs. I've// cut these out.if (params == null) {throw new IllegalArgumentException("Parameters may not be null");}Socket socket = null;int timeout = params.getConnectionTimeout();if (timeout == 0) {socket = createSocket(host, port, localAddress, localPort);} else {socket = new Socket();InetAddress hostAddress;Thread current = Thread.currentThread();if (current instanceof HostResolver) {HostResolver resolver = (HostResolver)current;hostAddress = resolver.resolve(host);} else {hostAddress = null;}InetSocketAddress address = (hostAddress != null)?new InetSocketAddress(hostAddress, port):new InetSocketAddress(host, port);socket.bind(new InetSocketAddress(localAddress, localPort));try {socket.connect(address, timeout);} catch (SocketTimeoutException e) {// Add timeout info. to the exception.throw new SocketTimeoutException(e.getMessage() +": timeout set at " + Integer.toString(timeout) + "ms.");}assert socket.isConnected(): "Socket not connected " + host;}return socket;}@Overridepublic Socket createSocket(String host, int port) throws IOException,UnknownHostException {// TODO Auto-generated method stubreturn new Socket(host, port);}/*** All instances of DefaultProtocolSocketFactory are the same.* @param obj Object to compare.* @return True if equal*/public boolean equals(Object obj) {return ((obj != null) &&obj.getClass().equals(HeritrixProtocolSocketFactory.class));}/*** All instances of DefaultProtocolSocketFactory have the same hash code.* @return Hash code for this object.*/public int hashCode() {return HeritrixProtocolSocketFactory.class.hashCode();}}HeritrixSSLProtocolSocketFactory类通过SecureProtocolSocketFactory实现SecureProtocolSocketFactory接⼝(间接实现了ProtocolSocketFactory接⼝)⽤于HTTPS通信SecureProtocolSocketFactory接⼝⽅法如下/*** A ProtocolSocketFactory that is secure.** @see mons.httpclient.protocol.ProtocolSocketFactory** @author Michael Becke* @author <a href="mailto:mbowler@">Mike Bowler</a>* @since 2.0*/public interface SecureProtocolSocketFactory extends ProtocolSocketFactory {/*** Returns a socket connected to the given host that is layered over an* existing socket. Used primarily for creating secure sockets through* proxies.** @param socket the existing socket* @param host the host name/IP* @param port the port on the host* @param autoClose a flag for closing the underling socket when the created* socket is closed** @return Socket a new socket** @throws IOException if an I/O error occurs while creating the socket* @throws UnknownHostException if the IP address of the host cannot be* determined*/Socket createSocket(Socket socket,String host,int port,boolean autoClose) throws IOException, UnknownHostException;}HeritrixSSLProtocolSocketFactory类实现上⾯的SecureProtocolSocketFactory接⼝/*** Implementation of the commons-httpclient SSLProtocolSocketFactory so we* can return SSLSockets whose trust manager is* {@link org.archive.httpclient.ConfigurableX509TrustManager}.** We also go to the heritrix cache to get IPs to use making connection.* To this, we have dependency on {@link HeritrixProtocolSocketFactory};* its assumed this class and it are used together.* See {@link HeritrixProtocolSocketFactory#getHostAddress(ServerCache,String)}.** @author stack* @version $Id: HeritrixSSLProtocolSocketFactory.java 6637 2009-11-10 21:03:27Z gojomo $ * @see org.archive.httpclient.ConfigurableX509TrustManager*/public class HeritrixSSLProtocolSocketFactory implements SecureProtocolSocketFactory { // static final String SERVER_CACHE_KEY = "heritrix.server.cache";static final String SSL_FACTORY_KEY = "heritrix.ssl.factory";/**** Socket factory with default trust manager installed.*/private SSLSocketFactory sslDefaultFactory = null;/*** Shutdown constructor.* @throws KeyManagementException* @throws KeyStoreException* @throws NoSuchAlgorithmException*/public HeritrixSSLProtocolSocketFactory()throws KeyManagementException, KeyStoreException, NoSuchAlgorithmException{// Get an SSL context and initialize it.SSLContext context = SSLContext.getInstance("SSL");// I tried to get the default KeyManagers but doesn't work unless you// point at a physical keystore. Passing null seems to do the right// thing so we'll go w/ that.context.init(null, new TrustManager[] {new ConfigurableX509TrustManager(ConfigurableX509TrustManager.DEFAULT)}, null);this.sslDefaultFactory = context.getSocketFactory();}@Overridepublic Socket createSocket(String host, int port, InetAddress clientHost,int clientPort)throws IOException, UnknownHostException {return this.sslDefaultFactory.createSocket(host, port,clientHost, clientPort);}@Overridepublic Socket createSocket(String host, int port)throws IOException, UnknownHostException {return this.sslDefaultFactory.createSocket(host, port);}@Overridepublic synchronized Socket createSocket(String host, int port,InetAddress localAddress, int localPort, HttpConnectionParams params)throws IOException, UnknownHostException {// Below code is from the DefaultSSLProtocolSocketFactory#createSocket// method only it has workarounds to deal with pre-1.4 JVMs. I've// cut these out.if (params == null) {throw new IllegalArgumentException("Parameters may not be null");}Socket socket = null;int timeout = params.getConnectionTimeout();if (timeout == 0) {socket = createSocket(host, port, localAddress, localPort);} else {SSLSocketFactory factory = (SSLSocketFactory)params.getParameter(SSL_FACTORY_KEY);//SSL_FACTORY_KEYSSLSocketFactory f = (factory != null)? factory: this.sslDefaultFactory;socket = f.createSocket();Thread current = Thread.currentThread();InetAddress hostAddress;if (current instanceof HostResolver) {HostResolver resolver = (HostResolver)current;hostAddress = resolver.resolve(host);} else {hostAddress = null;}InetSocketAddress address = (hostAddress != null)?new InetSocketAddress(hostAddress, port):new InetSocketAddress(host, port);socket.bind(new InetSocketAddress(localAddress, localPort));try {socket.connect(address, timeout);} catch (SocketTimeoutException e) {// Add timeout info. to the exception.throw new SocketTimeoutException(e.getMessage() +": timeout set at " + Integer.toString(timeout) + "ms.");}assert socket.isConnected(): "Socket not connected " + host;}return socket;}@Overridepublic Socket createSocket(Socket socket, String host, int port,boolean autoClose)throws IOException, UnknownHostException {return this.sslDefaultFactory.createSocket(socket, host,port, autoClose);}public boolean equals(Object obj) {return ((obj != null) && obj.getClass().equals(HeritrixSSLProtocolSocketFactory.class));}public int hashCode() {return HeritrixSSLProtocolSocketFactory.class.hashCode();}}HTTPS通信的SOCKET对象是通过SSLSocketFactory sslDefaultFactory（SSLSocket⼯⼚）对象创建的，为了创建SSLSocketFactory sslDefaultFactory对象Heritrix3.1.0系统定义了X509TrustManager接⼝的实现类ConfigurableX509TrustManager（⽤于SSL通信，⾃动接收证书）/*** A configurable trust manager built on X509TrustManager.** If set to 'open' trust, the default, will get us into sites for whom we do* not have the CA or any of intermediary CAs that go to make up the cert chain* of trust. Will also get us past selfsigned and expired certs. 'loose'* trust will get us into sites w/ valid certs even if they are just* selfsigned. 'normal' is any valid cert not including selfsigned. 'strict'* means cert must be valid and the cert DN must match server name.** <p>Based on pointers in* <a href="/commons/httpclient/sslguide.html">SSL* Guide</a>,* and readings done in <a* href="/j2se/1.4.2/docs/guide/security/jsse/JSSERefGuide.html#Introduction">JSSE* Guide</a>.** <p>TODO: Move to an ssl subpackage when we have other classes other than* just this one.** @author stack* @version $Id: ConfigurableX509TrustManager.java 6637 2009-11-10 21:03:27Z gojomo $*/public class ConfigurableX509TrustManager implements X509TrustManager{/*** Logging instance.*/protected static Logger logger = Logger.getLogger("org.archive.httpclient.ConfigurableX509TrustManager");public static enum TrustLevel {/*** Trust anything given us.** Default setting.** <p>See <a href="/egs/.ssl/TrustAll.html">* e502. Disabling Certificate Validation in an HTTPS Connection</a> from* the java almanac for how to trust all.*/OPEN,/*** Trust any valid cert including self-signed certificates.*//*** Normal jsse behavior.** Seemingly any certificate that supplies valid chain of trust.*/NORMAL,/*** Strict trust.** Ensure server has same name as cert DN.*/STRICT,}/*** Default setting for trust level.*/public final static TrustLevel DEFAULT = TrustLevel.OPEN;/*** Trust level.*/private TrustLevel trustLevel = DEFAULT;/*** An instance of the SUNX509TrustManager that we adapt variously* depending upon passed configuration.** We have it do all the work we don't want to.*/private X509TrustManager standardTrustManager = null;public ConfigurableX509TrustManager()throws NoSuchAlgorithmException, KeyStoreException {this(DEFAULT);}/*** Constructor.** @param level Level of trust to effect.** @throws NoSuchAlgorithmException* @throws KeyStoreException*/public ConfigurableX509TrustManager(TrustLevel level)throws NoSuchAlgorithmException, KeyStoreException {super();TrustManagerFactory factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());// Pass in a null (Trust) KeyStore. Null says use the 'default'// 'trust' keystore (KeyStore class is used to hold keys and to hold// 'trusts' (certs)). See 'X509TrustManager Interface' in this doc://// /j2se/1.4.2/docs/guide/security/jsse/JSSERefGuide.html#Introduction factory.init((KeyStore)null);TrustManager[] trustmanagers = factory.getTrustManagers();if (trustmanagers.length == 0) {throw new NoSuchAlgorithmException(TrustManagerFactory.getDefaultAlgorithm() + " trust manager not supported");}this.standardTrustManager = (X509TrustManager)trustmanagers[0];this.trustLevel = level;}@Overridepublic void checkClientTrusted(X509Certificate[] certificates, String type) throws CertificateException {if (this.trustLevel.equals(TrustLevel.OPEN)) {return;}this.standardTrustManager.checkClientTrusted(certificates, type);}@Overridepublic void checkServerTrusted(X509Certificate[] certificates, String type) throws CertificateException {if (this.trustLevel.equals(TrustLevel.OPEN)) {try {this.standardTrustManager.checkServerTrusted(certificates, type);if (this.trustLevel.equals(TrustLevel.STRICT)) {logger.severe(TrustLevel.STRICT + " not implemented.");}} catch (CertificateException e) {if (this.trustLevel.equals(TrustLevel.LOOSE) &&certificates != null && certificates.length == 1){// If only one cert and its valid and it caused a// CertificateException, assume its selfsigned.X509Certificate certificate = certificates[0];certificate.checkValidity();} else {// If we got to here, then we're probably NORMAL. Rethrow.throw e;}}}@Overridepublic X509Certificate[] getAcceptedIssuers() {return this.standardTrustManager.getAcceptedIssuers();}}---------------------------------------------------------------------------本系列Heritrix 3.1.0 源码解析系本⼈原创转载请注明出处博客园刺猬的温驯。

java爬虫框架有哪些，各有什么特点目前主流的Java爬虫框架主要有Nutch、Crawler4j、WebMagic、scrapy、WebCollector等，各有各的特点，大家可以根据自己的需求选择使用，下面为大家详细介绍常见的java爬虫框架有哪些？各有什么特点？常见的java爬虫框架有哪些1、NutchNutch是一个基于Lucene，类似Google的完整网络搜索引擎解决方案，基于Hadoop的分布式处理模型保证了系统的性能，类似Eclipse 的插件机制保证了系统的可客户化，而且很容易集成到自己的应用之中。

总体上Nutch可以分为2个部分：抓取部分和搜索部分。

抓取程序抓取页面并把抓取回来的数据做成反向索引，搜索程序则对反向索引搜索回答用户的请求。

抓取程序和搜索程序的接口是索引，两者都使用索引中的字段。

抓取程序和搜索程序可以分别位于不同的机器上。

下面详细介绍一下抓取部分。

Nutch抓取部分：抓取程序是被Nutch的抓取工具驱动的。

这是一组工具，用来建立和维护几个不同的数据结构：web database，a set of segments，and the index。

下面逐个解释这三个不同的数据结构：1、The web database，或者WebDB。

这是一个特殊存储数据结构，用来映像被抓取网站数据的结构和属性的集合。

WebDB 用来存储从抓取开始（包括重新抓取）的所有网站结构数据和属性。

WebDB 只是被抓取程序使用，搜索程序并不使用它。

WebDB 存储2种实体：页面和链接。

页面表示网络上的一个网页，这个网页的Url作为标示被索引，同时建立一个对网页内容的MD5 哈希签名。

跟网页相关的其它内容也被存储，包括：页面中的链接数量（外链接），页面抓取信息（在页面被重复抓取的情况下），还有表示页面级别的分数score 。

链接表示从一个网页的链接到其它网页的链接。

因此WebDB 可以说是一个网络图，节点是页面，链接是边。

Lucene很强大，这点在前面的章节中，已经作了详细介绍。

但是，无论多么强大的搜索引擎工具，在其后台，都需要一样东西来支援它，那就是网络爬虫Spi der。

网络爬虫，又被称为蜘蛛Spider，或是网络机器人、BOT等，这些都无关紧要，最重要的是要认识到，由于爬虫的存在，才使得搜索引擎有了丰富的资源。

Heritrix是一个纯由Java开发的、开源的Web网络爬虫，用户可以使用它从网络上抓取想要的资源。

它来自于。

Heritrix最出色之处在于它的可扩展性，开发者可以扩展它的各个组件，来实现自己的抓取逻辑。

本章就来详细介绍一下Heritrix和它的各个组件。

10.1 Heritrix的使用入门要想学会使用Heritrix，当然首先得能把它运行起来。

然而，运行Heritrix并非一件容易的事，需要进行很多配置。

在Heritrix的文档中对它的运行有详细的介绍，不过尽管如此，笔者仍然花了大量时间，才将其配置好并运行成功。

10.1.1 下载和运行HeritrixHeritrix的下载页面为：/downloads.html。

从上面可以链接到SourceForge的下载页面。

当前Heritrix的最新版本为1.10。

（1）在下载完Heritrix的完整开发包后，解压到本地的一个目录下，如图10-1所示。

图10-1 Heritrix的目录结构其中，Heritrix所用到的工具类库都存于lib下，heritrix-1.10.1.jar是Her itrix的Jar包。

另外，在Heritrix目录下有一个conf目录，其中包含了一个很重要的文件：heritrix.properties。

（2）在heritrix.properties中配置了大量与Heritrix运行息息相关的参数，这些参数主要是配置了Heritrix运行时的一些默认工具类、WebUI的启动参数，以及Heritrix的日志格式等。

当第一次运行Heritrix时，只需要修改该文件，为其加入WebUI的登录名和密码，如图10-2所示。