web crawler - Perl Mechanize module for scraping pdfs -

-

i have website many pdfs uploaded. want download pdfs present in website. first need provide username , password website. after searching sometime found www::mechanize package work. problem arises here want make recursive search in website meaning if link not contain pdf, should not discard link should navigate link , check whether new page has links contain pdfs. in way should exhaustively search entire website download pdfs uploaded. suggestion on how this?

i'd go wget, runs on variety of platforms.

if want in perl, check cpan web crawlers.

you might want decouple collecting pdf urls downloading them. crawling lengthy processing , might advantageous able hand off downloading tasks seperate worker processes.


Comments

Post a Comment

Popular posts from this blog

Cursor error with postgresql, pgpool and php -

-
hey, struggling bit determine exact cause of error has been popping in our release environment. there not seem dealing particular error on google. this error message getting: sqlstate[34000]: invalid cursor name: 7 error: portal "" not exist the error pops when using pdo prepared statements. this setup our release environment: pgpool 3.0.1 (the postgresql backend in streaming replication mode!) php 5.3.5 postgresql 9.0 edit: architecture 64bit. the same error not manifest in our test environment (edit: forgot mention, standard test environment uses postgresql 9.0 without pgpool). thus, led suspect pgpool @ least partly suspect. does know probable causes error are? edit: ok, here example of kind of code causes error. $sql = 'select * '; $sql .= 'from "mytable" "mystuff" '; $sql .= 'where "mytable"."status" = 1 '; $sql .= 'and "mytable"."mytableid" = :table...
Read more

delphi - ESC/P programming! -

-
why need use because printers using tprinter prints weird hieroglyph @ beginning of printable area. my problem if send commands, nothing happens. esc e (#27 #69) - sending escape sequence didn't work expected. removed first letter , rest of string made bold. eg. hello -> ello . after changed esc e esc (#27 #69 #27), worked fine. example managed figure out, but... trying select character table "esc t n" (#27 #116 n), make "õäöü" work. command doesn't work. nothing happens! , command supported esc/p, esc/p 2 , 9-pin esc/p, should work fine. manual can found here . if has ever needed use esc/p commands maybe 1 can shed light how work them! thanks in advance! edit: in previous post asked more or less same question though answers how did send commands printer. (i'll change question there according answers!) i accepted ken's answer, because claimed way transferred commands wasn't best , got work escape command. problem after t...
Read more

c++ - error: use of deleted function -

-
i've been working on c++ code friend has written , following error have never seen before when compiling gcc4.6: error: use of deleted function ‘gamefsm_<std::array<c, 2ul> >::hdealt::hdealt()’ implicitly deleted because default definition ill-formed: uninitialized non-static const member ‘const h_t floppokergamefsm_<std::array<c, 2ul> >::hdealt::h’ edit: comes part of code using boost msm: boost webpage edit2: there no = delete() used anywhere in sourcecode. generally speaking, error mean? should looking when type of error occurs? i don't think other answers mentioning =deleted; syntax correct. error message says default constructor has been deleted implicitly . says why: class contains non-static, const variable, not initialized default ctor. class x { const int x; }; since x::x const , must initialized -- default ctor wouldn't initialize (because it's pod type). therefore, default ctor, need define 1 (and must ...
Read more