This is worth a periodic re-share:
"Command-line Tools can be 235x Faster than your Hadoop Cluster"
Because sometimes the old ways are good ways. 🧓
@HerraBRE Here's another story along those lines: https://andre.arko.net/2018/10/25/parsing-logs-230x-faster-with-rust/
Goes from a Ruby script, to a $1000/month hi-tech AWS/Python/Hadoop solution, to a boring Rust program with CPU usage low enough to run for free on Lambda.
@HerraBRE This comparison is like saying that a paper and pen are better than programming because you can write “hello world” with paper and pen faster than writing and executing a program to do it. Hadoop is meant to get more performant as the data increases and is almost explicitly not meant to be used with such tiny amounts of data as this.
Generalistic and moderated instance. All opinions are welcome, but hate speeches are prohibited. Users who don't respect rules will be silenced or suspended, depending on the violation severity.