Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Did the authors benchmark against kdb?

e.g., run a single k interpeter on each CPU, then divide and conquer

Isn't it faster to do parsing in memory and avoid I/O wherever possible?



Best of three, on an i7 8GB ram:

Loading a 156mb csv file in kdb 32-bit free version, single thread:

    \t trade:`sym`time`ex`cond`size`price!("STCCXH";",")0:`t.csv
    1850
in paratext, 64-bit:

    >>> timeit.timeit('paratext.load_csv_to_dict("t.csv",num_threads=4)', setup="import paratext", number=1)
    3.1176819801330566
No improvement.

Loading a 1.5GB csv file in kdb:

    \t quote:`sym`time`ex`bid`bsize`ask`asize`mode!("STCHXHXC";",")0:`q.csv
    14135
in paratext:

    >>> timeit.timeit('paratext.load_csv_to_dict("q.csv",num_threads=4)', setup="import paratext", number=1)
    12.962939977645874
Not too shabby! An almost 10% improvement over KDB by turning my fans on and burning my lap!

However, I think they should probably make their parser faster before they waste heat trying to make slow code finish sooner.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: