Fastest Way to Parse a Large File in Ruby -

-

i have simple text file ~150mb. code read each line, , if matches regexes, gets written output file. right now, takes long time iterate through of lines of file (several minutes) doing 

file.open(filename).each |line|   # stuff end

i know looping through lines of file taking while because if nothing data in "#do stuff", still takes long time.

i know unix programs can parse large files instantly (like grep), wondering why ruby (mri 1.9) takes long read file, , there way make faster?

file.readlines.each |line|   #do stuff each line end

will read whole file 1 array of lines. should lot faster, takes more memory.


Comments

Post a Comment

Popular posts from this blog

c# - how to write client side events functions for the combobox items -

-
<dx:aspxcombobox id="aspxcombobox2" runat="server" clientidmode="autoid" selectedindex="-1" onselectedindexchanged="aspxcombobox1_selectedindexchanged1" > <clientsideevents selectedindexchanged="function(s, e) { grid.selectrows();}" /> <items> <dx:listedititem text="selecte rows" /> <dx:listedititem text="unselectall" /> </items> </dx:aspxcombobox> i want write event "unselect all" item in combobox. how can this? you'll need write javascript function iterates thru combo-box , marks selected items false. see select/unselect checkbox fields using pure javascript , html sample code msdn:
Read more

exception - Python, pyPdf OCR error: pyPdf.utils.PdfReadError: EOF marker not found -

-
pypdf throws exception: pypdf.utils.pdfreaderror: eof marker not found i don't need fix pypdf, need eof error cause "except" block execute , skip on file, doesn't work. still causes program stop running. background: batch ocr program pdfs python, pypdf, adobe pdf ocr error: unsupported filter /lzwdecode ... saga continues. i got 10,000 pdfs in folder. ocrd, not. can't tell 'em apart. step 1 figure out ones not ocrd , ocr (see other threads details). so i'm using pypdf. exceptions related unrecognized characters , unsupported filters when try read text. guestimated if throws exception, it's got text in , doens't go in list. problem solved, right? so: pypdf import pdffilewriter, pdffilereader import sys, os, pypdf, re path = 'c:\users\homer\documents\my pdfs' filelist = os.listdir(path) has_text_list = [] does_not_have_text_list = [] pdf_name in filelist: pdf_file
Read more