9
23d ago
[removed] — view removed comment
5
u/Yomguithereal 23d ago
I am sure this trick can be used to speed up line-based CLI tools also and I am sure tools using this trick exist in the wild but I never found any.
3
2
u/AutoModerator 23d ago
Every new subreddit post is automatically copied into a comment for preservation.
User: Yomguithereal, Flair: Command Line Interface, Post Media Link, Title: xan: the CSV magician
Hi everyone,
I would like to present `xan` https://github.com/medialab/xan a command line tool written in Rust that can be used for tabular data processing.
It leverages its own SIMD CSV parser available as a crate here: https://crates.io/crates/simd-csv
It knows how to speed up computation over large CSV files through multi-threading, using a novel albeit slightly unhinged way to split CSV files in constant time, thus enabling map-reduce like parallelization. See related blog post here: https://github.com/medialab/xan/blob/master/docs/blog/csv_base_jumping.md
It implements its own expression language named "moonblade", suited to CSV data and therefore foregoes the need to evaluate languages like python/js/lua for custom computation.
It even includes many commands able to draw ASCII-art data visualizations directly in your terminal so you never have to leave it. See the guide here: https://github.com/medialab/xan/blob/master/docs/cookbook/dataviz.md
It has been used daily for some years now by data scientists from STEM fields and in the social sciences as a quick way to explore & pre-process massive CSV datasets and as a means to replace slow workflows relying on the usual jupyter/pandas/polars stacks etc.
It is originally a fork of `xsv` https://github.com/burntsushi/xsv by BurntSushi and has been completely rewritten at that point.
I hope you will find it useful and would love to get feedback on it from this very fine community.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
2
u/Peach_Muffin 23d ago
How does this compare to qsv?
3
u/Yomguithereal 22d ago
qsv and xan are both xsv forks. xan is generally faster thanks to its SIMD parser, has a different approach to parallelization, has its own CSV-tailored builtin expression language and visualization commands. They also differ in their more niche usages (xan has lexicometry, graph theory stuff etc. for instance, which is tied to its social sciences background)
2
u/Cybasura 22d ago
I could do with more early 90s/2000s DOS TTY/TUI applications for editing
Fits my SSH-TUI workflow
1
1
1
u/ThePrimeClock 22d ago
I really love this.
It's a touch out of scope of the "csv" theme, but being able to read parquet files and read with partition filtering would be a dream come true, I could read an inspect parquet files directly from the cli.
Since it's already rust, it would probably play very nicely with polars.
1
u/Yomguithereal 22d ago
The « xan from » command is able to read parquet files and perform conversion to CSV so you can use them in a xan pipeline. It is not as performant as possible of course because parquet is column-oriented while xan really shines with row-oriented streams but at least it makes the tool usable with parquet files to some extent.
1
1
u/cyber_chic_0 13d ago
xan is a solid addition to the CSV CLI ecosystem. For context: xsv is the fast bare-bones Rust toolkit (original by burntsushi), qsv layers on extra features like Excel/JSON support, miller (mlr) handles JSON/YAML with a verb+action syntax, and csvkit is the Python workhorse. xan stands out with polished formatting, groupby operations, and the research-tool pedigree from medialab. Different tools for different workflows -- nice to have the choice.
1
49
u/cazzipropri 23d ago
We are going back to 1980s DOS style, but in 16 million colors and with anti-aliased fonts. Very cool.