http://tech.groups.yahoo.com/group/archive-crawler/ archive-crawler Sun, 08 Nov 2009 22:05:18 GMT Matthew Warhaftig http://tech.groups.yahoo.com/group/archive-crawler/message/6140 http://tech.groups.yahoo.com/group/archive-crawler/message/6140 Hi, In H3 I am trying to setup crawl jobs that use FetchHistoryProcessor/ PersistStoreProcessor/PersistLoadProcessor to discard duplicate content. I can get Tue, 03 Nov 2009 00:06:07 GMT pbaclace http://tech.groups.yahoo.com/group/archive-crawler/message/6139 http://tech.groups.yahoo.com/group/archive-crawler/message/6139 After some testing I determined that conf/profiles is created lazily if either a new profile is created or if the default profile is edited in the web UI. To Mon, 02 Nov 2009 12:40:07 GMT olintocattaneo http://tech.groups.yahoo.com/group/archive-crawler/message/6138 http://tech.groups.yahoo.com/group/archive-crawler/message/6138 Replying to myself just in case anyone competent missed this. Olinto Sat, 31 Oct 2009 18:12:24 GMT parseram34 http://tech.groups.yahoo.com/group/archive-crawler/message/6137 http://tech.groups.yahoo.com/group/archive-crawler/message/6137 I was reebooting and now the http://127.0.0.1:8080 address shows "Failed to connect". http://localhost:8080 doesnt work either. When I start the terminal its Fri, 30 Oct 2009 16:06:18 GMT Søren Vejrup Carlsen http://tech.groups.yahoo.com/group/archive-crawler/message/6136 http://tech.groups.yahoo.com/group/archive-crawler/message/6136 Hi Gordon. I can't find the tool to migrate 1.X configurations to 3.X style configurations. I have downloaded the heritrix-3.0.0-beta-dist.tar.gz from Thu, 29 Oct 2009 19:36:07 GMT Gordon Mohr http://tech.groups.yahoo.com/group/archive-crawler/message/6135 http://tech.groups.yahoo.com/group/archive-crawler/message/6135 When and where does this error appear? (For example: at the time Heritrix is launched, at the time you try to start a crawl, at the time you edit settings, Thu, 29 Oct 2009 10:02:17 GMT parseram34 http://tech.groups.yahoo.com/group/archive-crawler/message/6134 http://tech.groups.yahoo.com/group/archive-crawler/message/6134 Could you please help me..I get the following Error message in Heritrix: Wrong document type 'Crawl-order' in Thu, 29 Oct 2009 00:55:29 GMT Gordon Mohr http://tech.groups.yahoo.com/group/archive-crawler/message/6133 http://tech.groups.yahoo.com/group/archive-crawler/message/6133 The 'queued' URIs are almost certainly on some hosts that are not responding. Heritrix is trying them every 15 minutes, but then putting them back on the queue Wed, 28 Oct 2009 14:46:44 GMT Pranay Pandey http://tech.groups.yahoo.com/group/archive-crawler/message/6132 http://tech.groups.yahoo.com/group/archive-crawler/message/6132 Hi, I had set up a crawl job to run for 6 hours using H3-beta. I had it configured to be least polite and number of parallel queue was set to 5. After Tue, 27 Oct 2009 16:16:09 GMT steve@... http://tech.groups.yahoo.com/group/archive-crawler/message/6131 http://tech.groups.yahoo.com/group/archive-crawler/message/6131 hi Roger, the latest versions of Heritrix deliver warc output in format: "WARC File Format 1.0" which conforms to the ISO 28500 specification, an ISO standard Tue, 27 Oct 2009 16:05:58 GMT Coram, Roger http://tech.groups.yahoo.com/group/archive-crawler/message/6130 http://tech.groups.yahoo.com/group/archive-crawler/message/6130 Hi, Is there any documentation on the Heritrix implementation of WARC beyond just the source code? i.e. elements from the specification in-/excluded, which Tue, 27 Oct 2009 16:05:39 GMT steve@... http://tech.groups.yahoo.com/group/archive-crawler/message/6129 http://tech.groups.yahoo.com/group/archive-crawler/message/6129 hi Tram, can you be more specific about what you consider "junk"? the default profile includes a TransclusionDecideRule which tells the crawler to transitively Tue, 27 Oct 2009 15:44:08 GMT alphonse.smith16 http://tech.groups.yahoo.com/group/archive-crawler/message/6128 http://tech.groups.yahoo.com/group/archive-crawler/message/6128 Hello all, I am new on this group. I am looking for a web crawler which can get list of links of a webpage and convert a website downloaded in mht file. It Fri, 23 Oct 2009 02:04:50 GMT tristram.bethea http://tech.groups.yahoo.com/group/archive-crawler/message/6127 http://tech.groups.yahoo.com/group/archive-crawler/message/6127 Hello guys, I've been using heritrix to do some crawls with about 10 seeds. I find that I am getting excessively large amounts of trash in the data I collect. Thu, 22 Oct 2009 20:31:58 GMT olintocattaneo http://tech.groups.yahoo.com/group/archive-crawler/message/6126 http://tech.groups.yahoo.com/group/archive-crawler/message/6126 Hello I'm trying to get QueueOverbudgetDecideRule to work but I don't seem to be able to do this. Is this module still functional or maybe I have added it to a