e.g., run a single k interpeter on each CPU, then divide and conquer
Isn't it faster to do parsing in memory and avoid I/O wherever possible?
Loading a 156mb csv file in kdb 32-bit free version, single thread:
\t trade:`sym`time`ex`cond`size`price!("STCCXH";",")0:`t.csv 1850
>>> timeit.timeit('paratext.load_csv_to_dict("t.csv",num_threads=4)', setup="import paratext", number=1) 3.1176819801330566
Loading a 1.5GB csv file in kdb:
\t quote:`sym`time`ex`bid`bsize`ask`asize`mode!("STCHXHXC";",")0:`q.csv 14135
>>> timeit.timeit('paratext.load_csv_to_dict("q.csv",num_threads=4)', setup="import paratext", number=1) 12.962939977645874
However, I think they should probably make their parser faster before they waste heat trying to make slow code finish sooner.
e.g., run a single k interpeter on each CPU, then divide and conquer
Isn't it faster to do parsing in memory and avoid I/O wherever possible?