.net - Parallel Foreach Memory Issue -

-

i have file collection (3000 files) in fileinfocollection. want process files applying logic independent (can executed in parallel).

 fileinfo[] fileinfocollection = directory.getfiles();  parallel.foreach(fileinfocollection, processworkeritem);

but after processing 700 files getting out of memory error. used thread-pool before giving same error. if try execute without threading (parallel processing) works fine.

in "processworkeritem" running algorithm based on string data of file. additionally use log4net logging , there lot of communications sql server in method.

here info, files size : 1-2 kb xml files. read files , process dependent on content of file. identifying keywords in string , generating xml format. keywords in sql server database (nearly 2000 words).

well, processworkeritem do? may able change use less memory (e.g. stream data instead of loading in @ once) or may want explicitly limit degree of parallelism using this overload , paralleloptions.maxdegreeofparallelism. want avoid trying process 3000 files @ once :) iirc, parallel extensions "notice" if tasks appear io bound, , allow more normal number execute @ once - isn't want here, you're memory bound well.


Comments

Post a Comment

Popular posts from this blog

c# - how to write client side events functions for the combobox items -

-
<dx:aspxcombobox id="aspxcombobox2" runat="server" clientidmode="autoid" selectedindex="-1" onselectedindexchanged="aspxcombobox1_selectedindexchanged1" > <clientsideevents selectedindexchanged="function(s, e) { grid.selectrows();}" /> <items> <dx:listedititem text="selecte rows" /> <dx:listedititem text="unselectall" /> </items> </dx:aspxcombobox> i want write event "unselect all" item in combobox. how can this? you'll need write javascript function iterates thru combo-box , marks selected items false. see select/unselect checkbox fields using pure javascript , html sample code msdn:
Read more

exception - Python, pyPdf OCR error: pyPdf.utils.PdfReadError: EOF marker not found -

-
pypdf throws exception: pypdf.utils.pdfreaderror: eof marker not found i don't need fix pypdf, need eof error cause "except" block execute , skip on file, doesn't work. still causes program stop running. background: batch ocr program pdfs python, pypdf, adobe pdf ocr error: unsupported filter /lzwdecode ... saga continues. i got 10,000 pdfs in folder. ocrd, not. can't tell 'em apart. step 1 figure out ones not ocrd , ocr (see other threads details). so i'm using pypdf. exceptions related unrecognized characters , unsupported filters when try read text. guestimated if throws exception, it's got text in , doens't go in list. problem solved, right? so: pypdf import pdffilewriter, pdffilereader import sys, os, pypdf, re path = 'c:\users\homer\documents\my pdfs' filelist = os.listdir(path) has_text_list = [] does_not_have_text_list = [] pdf_name in filelist: pdf_file
Read more