bixo-dev at Yahoo! Groups

Latest updates in trunk/master

Ken Krugler — Tue, 05 Jan 2010 17:08:28 GMT

Hi all, I just pushed a set of major changes to trunk. The biggest change was new queue management during robots.txt processing and fetching, to avoid OOM

Re: HTTP 204

Fuad Efendi — Mon, 04 Jan 2010 03:06:19 GMT

int httpStatus = response.getStatusLine().getStatusCode(); if (httpStatus != HttpStatus.SC_OK) { throw new HttpFetchException(url, "Error fetching " + url,

Re: HTTP 204

Fuad Efendi — Mon, 04 Jan 2010 02:34:58 GMT

Hi Ken, Yes, I am using old (1-month probably) code from trunk; I did a lot of changes and can't easily move to latest version. My changes include additional

Re: HTTP 204

Ken Krugler — Sun, 03 Jan 2010 23:36:43 GMT

Hi Fuad, I've updated the code to avoid throwing an exception, but this indicates a potential problem someplace else. The HttpFetchException.mapToUrlStatus()

Re: Fetching 242 URLs from 0 domains (251646 URLs remaining)

Ken Krugler — Sat, 02 Jan 2010 22:46:32 GMT

Hi Fuad, I just tightened up some of the code that increments/decrements status counters, to catch some cases where exceptions would cause the counters to not

HTTP 204

Fuad Efendi — Sat, 02 Jan 2010 18:30:38 GMT

I was running fetch (3rd loop) over 24 hours, and finally... RuntimeException, job failed: [HttpFetchException] default: if (_httpStatus < 300) { throw new

Re: OutOfMemoryError

Fuad Efendi — Thu, 31 Dec 2009 17:25:13 GMT

Hi Ken, My use case: 3 reducers, over 30k hosts (including subdomains). So that I was forced to set in-memory limit 10 for DiskQueue, and now to limit

Re: OutOfMemoryError

Ken Krugler — Thu, 31 Dec 2009 16:02:30 GMT

Hi Fuad, I've got a version I'm testing now that returns false if all of the threads are active. Which means that in the case you mention (every site has some

Re: OutOfMemoryError

Fuad Efendi — Thu, 31 Dec 2009 15:00:41 GMT

OOM Disappeared after I implemented this (draft; FetcherQueueMgr): public boolean offer(FetcherQueue newQueue) { if (_queues.size()>100) return false;

Re: OutOfMemoryError

Fuad Efendi — Tue, 29 Dec 2009 15:30:10 GMT

I believe OOM still happens because FetcherQueueMgr.offer() is not fully implemented yet; I started crawl with 10k domains and 3 reducers, using IP for

OutOfMemoryError

Fuad Efendi — Tue, 29 Dec 2009 15:05:41 GMT

I am again with java.lang.OutOfMemoryError (Fetcher/Reducer) After implementing DiskQueue (in my environment), I run 2 first iterations without any problems,

Re: New domain name

bruno_abitbol — Wed, 23 Dec 2009 08:37:40 GMT

Hi Ken, personally I would prefer bixominer.org because it sounds good and tells what it does. Bruno.

New domain name

Ken Krugler — Wed, 23 Dec 2009 00:07:31 GMT

We're trying to pick a domain name to use for the Bixo project. Current options are: * bixo-project.org * openbixo.org * bixominer.org Any input and/or

Re: Beginner Question

Ken Krugler — Tue, 22 Dec 2009 15:39:30 GMT

Hi Bruno, ... You could use the LoadUrlsFunction with an Each() operator to import the URLs from a text file. For example, here's some code from a white-list

Re: Beginner Question

bruno_abitbol — Tue, 22 Dec 2009 15:17:15 GMT

Hi Ken, thank you for your quick response. ... How can I inject the 50 URLS in the crawler? ... 10000 to 50000 URLS per domain so let's say a total of 1 500