bixo-dev at Yahoo! Groups

Re: Advise for fetcher policy

Ken Krugler — Wed, 22 May 2013 17:10:50 GMT

... A 404 should result in UrlStatus.HTTP_NOT_FOUND What you do with those entries in the crawlDB is up to your processing code. In the DemoCrawlTool, the

Re: Advise for fetcher policy

Pat Ferrel — Wed, 22 May 2013 16:55:51 GMT

The miner is getting urls to 404s. So Pinterest is allowing removal of pages but leaving the links to those pages around. If leaving them in the crawldb marked

Re: Advise for fetcher policy

Ken Krugler — Wed, 22 May 2013 16:35:48 GMT

... One other thought - we do "batch" fetching of URLs, using keep-alive to optimize the connection that we create with the server. Pinterest might not like

Re: Advise for fetcher policy

Pat Ferrel — Wed, 22 May 2013 15:06:28 GMT

Again, thanks. You are the only one I know with crawler-fu skills. I'll take a look at the headers in the simple fetcher. I suspect that Pinterest is not

Re: Advise for fetcher policy

Ken Krugler — Wed, 22 May 2013 13:53:03 GMT

Hi Pat, ... In bixo there's a FetchAndParseTool that you can use to fetch individual URLs. I'd try that, as another way to test. Some random ideas?

Re: Advise for fetcher policy

Pat Ferrel — Tue, 21 May 2013 23:58:15 GMT

Thanks. With the below settings I also changed the retry to 0 and the log level to trace. It looks like perfectly good urls are not getting fetched. Thinking I

Re: Advise for fetcher policy

Ken Krugler — Fri, 17 May 2013 22:00:13 GMT

Hi Pat, ... It will try to fetch every URL, but it will only make one HttpClient request for each URL. HttpClient will retry multiple times, and if the server

Advise for fetcher policy

Pat Ferrel — Fri, 17 May 2013 14:00:08 GMT

Hi guys, I'm back to crawling Pinterest to update my experimental recommender. I created a merged miner/crawler, which was working fine if slowly. I added an

Re: building bixo

Pat Ferrel — Thu, 09 May 2013 18:00:37 GMT

Hmm, comment out the test and it completes without errors. Maybe openDNS is the problem? On May 9, 2013, at 10:31 AM, Pat Ferrel wrote:

building bixo

Pat Ferrel — Thu, 09 May 2013 17:31:10 GMT

It's been awhile since I did a new build of bixo. For some reason, though I haven't changed the code, I'm getting all sorts of test errors. I was getting an

Re: democrawler - domain

Ken Krugler — Thu, 04 Apr 2013 23:41:08 GMT

Hi Jeff, ... By default if you provide a -domain parameter, then URL filtering is set up such that only URLs for that domain are accepted (all other URLs are

Double-normalization of URLs?

Ken Krugler — Thu, 04 Apr 2013 23:36:23 GMT

Hi Vivek, I was looking at the DemoCrawlWorkflow source, and noticed this snippet: Pipe urlFromOutlinksPipe = new Pipe("url from outlinks",

democrawler - domain

jeffjeffrsn — Tue, 02 Apr 2013 17:22:07 GMT

Hi Eeryone, I noticed, that the democrawler stays at one domain. ... I've got the domain example.com. At this domain there are outlinks to test.example.com,

Re: html content - parsePipe.getTailPipe() (DemoCrawlTool)

jeffjeffrsn — Tue, 02 Apr 2013 17:02:54 GMT

Hi Chris, Thanks for the answer. Now I'm subclassing the baseparser. Thanks, - Jeff

Re: html content - parsePipe.getTailPipe() (DemoCrawlTool)

Chris Schneider — Tue, 02 Apr 2013 14:12:42 GMT

Hi Jeff, I am not sure what you meant when you wrote "added a new Pipe to the tail of the parsePipe". If you did add a tail pipe containing only