archive-crawler at Yahoo! Groups

about dns

郭芸 — Wed, 10 Feb 2010 03:12:31 GMT

Hi, why heritrix run for some time,it can not get any new url ,because it has been running dns?

Re: Updating crawler-beans.cxml via REST API

Daniel Truemper — Mon, 08 Feb 2010 16:39:03 GMT

Hi! ... This is very strange! I have again debugged through the code and as far as I can see, there should not be any reason for the rename to fail. The

Re: Budget & reporting on exhausted and retired queues

Gordon Mohr — Fri, 05 Feb 2010 20:38:36 GMT

... It appears you're using H1. Yes, queues with zero 'currentSize' in the 'all' report are empty (aka 'exhausted'). Queues with 'totalSpend/totalBudget' with

Budget & reporting on exhausted and retired queues

Nicolas Giraud — Fri, 05 Feb 2010 11:21:52 GMT

Hi, I have browsed the code in WorkQueueFrontier, and had a look at the JMX service method frontierReport. I would like to know how I could obtain a report

Re: H3: Severe Problem or misconfiguration

t.schoellhorn — Fri, 05 Feb 2010 08:57:47 GMT

Dear Gordon, I just checked my disks and they seemed to be fine. Now I am running that crawl again and will report if this error can be reproduced. It might

Re: Heritrix traffic pattern over time

Derek Pappas — Fri, 05 Feb 2010 05:15:07 GMT

$ uname -a Linux 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux $ java -version java version "1.6.0" OpenJDK Runtime

Re: Heritrix traffic pattern over time

Gordon Mohr — Fri, 05 Feb 2010 01:06:48 GMT

... Actually, they still aren't coming through (to me or in the archives - ). But, let's just

Re: H1 Checkpointing Re: [archive-crawler] Misc questions

Derek Pappas — Thu, 04 Feb 2010 23:45:18 GMT

... Will if the problem reoccurs. Thanks, Derek

Re: Heritrix traffic pattern over time

Derek Pappas — Thu, 04 Feb 2010 23:43:31 GMT

Here are the images that you could not see in the other post. Below is the toe-thread report. There are quite a few threads with step: ABOUT_TO_BEGIN_PROCESSOR

H1 Checkpointing Re: [archive-crawler] Misc questions

Gordon Mohr — Thu, 04 Feb 2010 23:00:54 GMT

From another thread: ... If you have a reproduceable hang, or time to investigate an intermittent occurrence, it would be useful to capture: - general

Re: how can I get the queued urls?

Gordon Mohr — Thu, 04 Feb 2010 22:55:47 GMT

... The exact techniques available depend on the version you're using. Some possibilities include: - a frontier 'dump-pending-at-close'/dumpPendingAtClose

Re: Heritrix traffic pattern over time

Gordon Mohr — Thu, 04 Feb 2010 19:39:24 GMT

FYI, the images did not display for me in this message; a paste of the text would be as useful. Swapping won't be an issue here, but it could still be: - a

Re: Heritrix traffic pattern over time

Derek Pappas — Thu, 04 Feb 2010 19:00:00 GMT

Here is another example on a quad core that started out at 16 URI's/ sec and is down to 8. heap used : 1.4 GB heap max/allocated: 1.8 GB RAM

Re: H3: Severe Problem or misconfiguration

Gordon Mohr — Thu, 04 Feb 2010 18:37:37 GMT

The SEVERE errors you're seeing suggest some corruption in the on-disk queues such that they don't have the expected structure. Most helpful in understanding

crawler-beans.cxml

t.schoellhorn — Thu, 04 Feb 2010 17:27:58 GMT

Here is my configuration which is a rather minor variation of the standard XML-File: