nutch

Question

这是crawl2.log里的出错的内容求解Fetcher: starting
Fetcher: segment: ../crawl2/segments/20100523211245
Fetcher: threads: 10
QueueFeeder finished: total 3 records.
fetching http://news.sina.com.cn/pfpnews/js/libweb.js
Error parsing: http://news.sina.com.cn/pfpnews/js/libweb.js: org.apache.nutch.parse.ParseException: parser not found for contentType=application/javascript url=http://news.sina.com.cn/pfpnews/js/libweb.js
at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:74)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:766)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:552)-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=2
* queue: http://121.194.0.210
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 1000
  minCrawlDelay = 0
  nextFetchTime = 1274620375828
  now           = 1274620375328
  0. http://news.sina.com.cn/js/213/20091202/34/head.js
  1. http://pfp.sina.com.cn/pfpnew/merge/res_PGLS000022_FP.js
fetching http://news.sina.com.cn/js/213/20091202/34/head.js
Error parsing: http://news.sina.com.cn/js/213/20091202/34/head.js: org.apache.nutch.parse.ParseException: parser not found for contentType=application/javascript url=http://news.sina.com.cn/js/213/20091202/34/head.js
at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:74)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:766)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:552)-activeThreads=10, spinWaiting=10, fetchQueues.totalSize=1
* queue: http://121.194.0.210
  maxThreads    = 1
  inProgress    = 0
  crawlDelay    = 1000
  minCrawlDelay = 0
  nextFetchTime = 1274620377296
  now           = 1274620376328
  0. http://pfp.sina.com.cn/pfpnew/merge/res_PGLS000022_FP.js
fetching http://pfp.sina.com.cn/pfpnew/merge/res_PGLS000022_FP.js
-finishing thread FetcherThread, activeThreads=9
-finishing thread FetcherThread, activeThreads=8
-finishing thread FetcherThread, activeThreads=7
-finishing thread FetcherThread, activeThreads=6
-finishing thread FetcherThread, activeThreads=5
-finishing thread FetcherThread, activeThreads=4
-finishing thread FetcherThread, activeThreads=3
-activeThreads=3, spinWaiting=2, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
Error parsing: http://pfp.sina.com.cn/pfpnew/merge/res_PGLS000022_FP.js: org.apache.nutch.parse.ParseException: parser not found for contentType=application/javascript url=http://pfp.sina.com.cn/pfpnew/merge/res_PGLS000022_FP.js
at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:74)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.output(Fetcher.java:766)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:552)-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, F.totalSize=0
-activeThreads=0
Fetcher: done
CrawlDb update: starting
CrawlDb update: db: ../crawl2/crawldb
CrawlDb update: segments: [../crawl2/segments/20100523211245]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: true
CrawlDb update: URL filtering: true
CrawlDb update: Merging segment data into db.
CrawlDb update: done
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: ../crawl2/segments/20100523211305
Generator: filtering: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=2 - no more URLs to fetch.
LinkDb: starting
LinkDb: linkdb: ../crawl2/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: file:/C:/cygwin/home/Administrator/nutch-1.0/crawl2/segments/20100523211222
LinkDb: adding segment: file:/C:/cygwin/home/Administrator/nutch-1.0/crawl2/segments/20100523211245
LinkDb: done
Indexer: starting
Indexer: done
Dedup: starting
Dedup: adding indexes in: ../crawl2/indexes
Dedup: done
merging indexes to: ../crawl2/index
Adding file:/C:/cygwin/home/Administrator/nutch-1.0/crawl2/indexes/part-00000
done merging
crawl finished: ../crawl2

Answer 1

你下载的网页全是.js，
fetching http://news.sina.com.cn/pfpnews/js/libweb.js
下载后解析内容出现异常，
Error parsing: 
所以之后没有真正建立索引。最后由于没有解析到url，还没爬完就终止了
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=2 - no more URLs to fetch.你的过滤文件中怎么设置的，怎么下载的全是.js文件，这些文件毫无意义啊？