.net - Parallel Foreach Memory Issue -

-

i have file collection (3000 files) in fileinfocollection. want process files applying logic independent (can executed in parallel).

 fileinfo[] fileinfocollection = directory.getfiles();  parallel.foreach(fileinfocollection, processworkeritem);

but after processing 700 files getting out of memory error. used thread-pool before giving same error. if try execute without threading (parallel processing) works fine.

in "processworkeritem" running algorithm based on string data of file. additionally use log4net logging , there lot of communications sql server in method.

here info, files size : 1-2 kb xml files. read files , process dependent on content of file. identifying keywords in string , generating xml format. keywords in sql server database (nearly 2000 words).

well, processworkeritem do? may able change use less memory (e.g. stream data instead of loading in @ once) or may want explicitly limit degree of parallelism using this overload , paralleloptions.maxdegreeofparallelism. want avoid trying process 3000 files @ once :) iirc, parallel extensions "notice" if tasks appear io bound, , allow more normal number execute @ once - isn't want here, you're memory bound well.


Comments

Post a Comment

Popular posts from this blog

Cursor error with postgresql, pgpool and php -

-
hey, struggling bit determine exact cause of error has been popping in our release environment. there not seem dealing particular error on google. this error message getting: sqlstate[34000]: invalid cursor name: 7 error: portal "" not exist the error pops when using pdo prepared statements. this setup our release environment: pgpool 3.0.1 (the postgresql backend in streaming replication mode!) php 5.3.5 postgresql 9.0 edit: architecture 64bit. the same error not manifest in our test environment (edit: forgot mention, standard test environment uses postgresql 9.0 without pgpool). thus, led suspect pgpool @ least partly suspect. does know probable causes error are? edit: ok, here example of kind of code causes error. $sql = 'select * '; $sql .= 'from "mytable" "mystuff" '; $sql .= 'where "mytable"."status" = 1 '; $sql .= 'and "mytable"."mytableid" = :table...
Read more

delphi - ESC/P programming! -

-
why need use because printers using tprinter prints weird hieroglyph @ beginning of printable area. my problem if send commands, nothing happens. esc e (#27 #69) - sending escape sequence didn't work expected. removed first letter , rest of string made bold. eg. hello -> ello . after changed esc e esc (#27 #69 #27), worked fine. example managed figure out, but... trying select character table "esc t n" (#27 #116 n), make "õäöü" work. command doesn't work. nothing happens! , command supported esc/p, esc/p 2 , 9-pin esc/p, should work fine. manual can found here . if has ever needed use esc/p commands maybe 1 can shed light how work them! thanks in advance! edit: in previous post asked more or less same question though answers how did send commands printer. (i'll change question there according answers!) i accepted ken's answer, because claimed way transferred commands wasn't best , got work escape command. problem after t...
Read more

c++ - error: use of deleted function -

-
i've been working on c++ code friend has written , following error have never seen before when compiling gcc4.6: error: use of deleted function ‘gamefsm_<std::array<c, 2ul> >::hdealt::hdealt()’ implicitly deleted because default definition ill-formed: uninitialized non-static const member ‘const h_t floppokergamefsm_<std::array<c, 2ul> >::hdealt::h’ edit: comes part of code using boost msm: boost webpage edit2: there no = delete() used anywhere in sourcecode. generally speaking, error mean? should looking when type of error occurs? i don't think other answers mentioning =deleted; syntax correct. error message says default constructor has been deleted implicitly . says why: class contains non-static, const variable, not initialized default ctor. class x { const int x; }; since x::x const , must initialized -- default ctor wouldn't initialize (because it's pod type). therefore, default ctor, need define 1 (and must ...
Read more