scala - Preferred way of processing this data with parallel arrays -

-

imagine sequence of java.io.file objects. sequence not in particular order, gets populated after directory traversal. names of files can this:

/some/file.bin /some/other_file_x1.bin /some/other_file_x2.bin /some/other_file_x3.bin /some/other_file_x4.bin /some/other_file_x5.bin ... /some/x_file_part1.bin /some/x_file_part2.bin /some/x_file_part3.bin /some/x_file_part4.bin /some/x_file_part5.bin ... /some/x_file_part10.bin

basically, can have 3 types of files. first type simple ones, have .bin extension. second type of file 1 formed _x1.bin till _x5.bin. , third type of file can formed of 10 smaller parts, _part1 till _part10. know naming may strange, have work :)

i want group files ( pieces of file should processed ), , thinking of using parallel arrays this. thing i'm not sure how can perform reduce/acumulation part, since threads working on same array.

val allbinfiles = allbins.toarray // array of java.io.file

i thinking of handling that:

val mapacumulator = java.util.collections.synchronizedmap[string,listbuffer[file]](new java.util.hashmap[string,listbuffer[file]]())  allbinfiles.par.foreach { file =>    file match {       // /some/x_file_x4.bin nametillpart /some/x_file       case composedof5name(nametillpart) => {           mapacumulator.getorelseupdate(nametillpart,new listbuffer[file]()) += file       }       case composedof10name(nametillpart) => {           mapacumulator.getorelseupdate(nametillpart,new listbuffer[file]()) += file       }       // simple file, without pieces       case _ => {           mapacumulator.getorelseupdate(file.tostring,new listbuffer[file]()) += file       }    } }

i thinking of doing i've shown in above code. having extractors files, , using part of path key in map. example, /some/x_file can hold values /some/x_file_x1.bin /some/x_file_x5.bin. think there better way of handling this. interested in opinions.

the alternative use groupby:

val mp = allbinfiles.par.groupby {   case composedof5name(x) => x   case composedof10name(x) => x   case f => f.tostring }

this return parallel map of parallel arrays of files (parmap[string, pararray[file]]). if want sequential map of sequential sequences of files point:

val sqmp = mp.map(_.seq).seq

to ensure parallelism kicks in, make sure have enough elements in parallel array (10k+).


Comments

Post a Comment

Popular posts from this blog

Cursor error with postgresql, pgpool and php -

-
hey, struggling bit determine exact cause of error has been popping in our release environment. there not seem dealing particular error on google. this error message getting: sqlstate[34000]: invalid cursor name: 7 error: portal "" not exist the error pops when using pdo prepared statements. this setup our release environment: pgpool 3.0.1 (the postgresql backend in streaming replication mode!) php 5.3.5 postgresql 9.0 edit: architecture 64bit. the same error not manifest in our test environment (edit: forgot mention, standard test environment uses postgresql 9.0 without pgpool). thus, led suspect pgpool @ least partly suspect. does know probable causes error are? edit: ok, here example of kind of code causes error. $sql = 'select * '; $sql .= 'from "mytable" "mystuff" '; $sql .= 'where "mytable"."status" = 1 '; $sql .= 'and "mytable"."mytableid" = :table...
Read more

delphi - ESC/P programming! -

-
why need use because printers using tprinter prints weird hieroglyph @ beginning of printable area. my problem if send commands, nothing happens. esc e (#27 #69) - sending escape sequence didn't work expected. removed first letter , rest of string made bold. eg. hello -> ello . after changed esc e esc (#27 #69 #27), worked fine. example managed figure out, but... trying select character table "esc t n" (#27 #116 n), make "õäöü" work. command doesn't work. nothing happens! , command supported esc/p, esc/p 2 , 9-pin esc/p, should work fine. manual can found here . if has ever needed use esc/p commands maybe 1 can shed light how work them! thanks in advance! edit: in previous post asked more or less same question though answers how did send commands printer. (i'll change question there according answers!) i accepted ken's answer, because claimed way transferred commands wasn't best , got work escape command. problem after t...
Read more

c++ - error: use of deleted function -

-
i've been working on c++ code friend has written , following error have never seen before when compiling gcc4.6: error: use of deleted function ‘gamefsm_<std::array<c, 2ul> >::hdealt::hdealt()’ implicitly deleted because default definition ill-formed: uninitialized non-static const member ‘const h_t floppokergamefsm_<std::array<c, 2ul> >::hdealt::h’ edit: comes part of code using boost msm: boost webpage edit2: there no = delete() used anywhere in sourcecode. generally speaking, error mean? should looking when type of error occurs? i don't think other answers mentioning =deleted; syntax correct. error message says default constructor has been deleted implicitly . says why: class contains non-static, const variable, not initialized default ctor. class x { const int x; }; since x::x const , must initialized -- default ctor wouldn't initialize (because it's pod type). therefore, default ctor, need define 1 (and must ...
Read more