http://tech.groups.yahoo.com/group/archive-crawler/ archive-crawler Fri, 20 Nov 2009 16:48:19 GMT stack http://tech.groups.yahoo.com/group/archive-crawler/message/6161 http://tech.groups.yahoo.com/group/archive-crawler/message/6161 Hey crawlers: I was Apachecon in Oakland in early November and was present during a meeting of a few of the open source crawler projects (Ken Krugle for Bixo, Thu, 19 Nov 2009 16:44:59 GMT Bjarne Andersen http://tech.groups.yahoo.com/group/archive-crawler/message/6160 http://tech.groups.yahoo.com/group/archive-crawler/message/6160 Our main challenge is that we need the queues sperated on TLD (foo.com and bar.com) to use the quota-enforcer to limit number of bytes on each TLD but at the Thu, 19 Nov 2009 16:08:34 GMT kristsi25 http://tech.groups.yahoo.com/group/archive-crawler/message/6159 http://tech.groups.yahoo.com/group/archive-crawler/message/6159 John pretty much sums up the problem. The way I've dealt with this has been on a case by case basis. Each time we detect this situation, we override the Thu, 19 Nov 2009 15:41:02 GMT John Lekashman http://tech.groups.yahoo.com/group/archive-crawler/message/6158 http://tech.groups.yahoo.com/group/archive-crawler/message/6158 Hi, It is not actually possible to guarantee this. There is no real way to distinguish for sure where the actual physical hardware that hosts a name is. There Thu, 19 Nov 2009 15:03:04 GMT Søren Vejrup Carlsen http://tech.groups.yahoo.com/group/archive-crawler/message/6157 http://tech.groups.yahoo.com/group/archive-crawler/message/6157 Hi all. Is it possible in Heritrix 1.14.3 to avoid overloading webservers hosting many virtual servers. We currently have the problem that those webservers Thu, 19 Nov 2009 08:01:02 GMT parseram34 http://tech.groups.yahoo.com/group/archive-crawler/message/6156 http://tech.groups.yahoo.com/group/archive-crawler/message/6156 I reinstalled Heritrix and got the same message again after configuring my first job: Wrong document type 'crawl-order' in Wed, 18 Nov 2009 15:57:02 GMT Tomas Ukkonen http://tech.groups.yahoo.com/group/archive-crawler/message/6155 http://tech.groups.yahoo.com/group/archive-crawler/message/6155 Hi Kris, Thank you for your reply. I will use JIRA as you suggested attach patches against the latest 1.14.x revision from the Heritrix repository. Regards, -- Tue, 17 Nov 2009 23:15:22 GMT Alexis http://tech.groups.yahoo.com/group/archive-crawler/message/6154 http://tech.groups.yahoo.com/group/archive-crawler/message/6154 Hi, IA is hiring a tech lead for our new web crawling initiative. You can read the job description here: http://www.archive.org/about/webjobs.php#wwcengineer Mon, 16 Nov 2009 11:27:36 GMT takeru sasaki http://tech.groups.yahoo.com/group/archive-crawler/message/6153 http://tech.groups.yahoo.com/group/archive-crawler/message/6153 Hi, I want to know about IDN support of heritrix. ("Internationalized domain name" http://en.wikipedia.org/wiki/Internationalized_Domain_Name) I was tried to Mon, 16 Nov 2009 10:38:47 GMT kristsi25 http://tech.groups.yahoo.com/group/archive-crawler/message/6152 http://tech.groups.yahoo.com/group/archive-crawler/message/6152 I'd suggest opening up an issue (probably multiple issues in your case) in the crawler issue base (https://webarchive.jira.com/browse/HER) and to then attach Sat, 14 Nov 2009 02:07:16 GMT Matthew Warhaftig http://tech.groups.yahoo.com/group/archive-crawler/message/6151 http://tech.groups.yahoo.com/group/archive-crawler/message/6151 Good advice, thank you Gordon. Adding the recrawl processors to the chain bean and pointing PersistLoadProcessor directly to my existing history (no preload) Fri, 13 Nov 2009 11:38:40 GMT Tomas Ukkonen http://tech.groups.yahoo.com/group/archive-crawler/message/6150 http://tech.groups.yahoo.com/group/archive-crawler/message/6150 Hi In National Library of Finland we have made some improvements to Heritrix: - document classification (content-based) deciderules support (e.g. Fri, 13 Nov 2009 10:21:07 GMT raffaele messuti http://tech.groups.yahoo.com/group/archive-crawler/message/6149 http://tech.groups.yahoo.com/group/archive-crawler/message/6149 ... not in java, but ruby: http://anemone.rubyforge.org/ just use from shell: $ anemone url-list http://crawler.archive.org > seeds.txt -- Thu, 12 Nov 2009 16:13:35 GMT pierce403 http://tech.groups.yahoo.com/group/archive-crawler/message/6148 http://tech.groups.yahoo.com/group/archive-crawler/message/6148 I am looking for a simple way to spider web pages from within an app I am working on. I know heritrix is not intended to be used as a library, but would using Tue, 10 Nov 2009 22:54:08 GMT Gordon Mohr http://tech.groups.yahoo.com/group/archive-crawler/message/6147 http://tech.groups.yahoo.com/group/archive-crawler/message/6147 Heritrix cannot execute Javascript, so its link-extraction with respect to Javascript uses a crude heuristic of trying strings that might be relative URIs