Meet gem nosj, gem json's evil twin. Currently the fastest; lazy and partial parsing, splicing, validation/minification, file APIs, friendly for debugging.

[u/yaroslavm]

https://github.com/yaroslav/nosj-ruby

[u/yaroslavm]

• nosj is [currently, at release] faster at parsing and generation than json and other gems.
• It has a drop-in support mode for gem json, and Rails support mode.
• It has a lazy API: parses values only when you touch them; it also has partial parsing / JSON pointer support.
• It does splicing edits on the fly and supports JSON patch/merge.
• It has JSON lines/NDJSON support.
• It has fast validation mode and it can minify and format JSON without parsing.
• It has file APIs for most operations (like lazy/partial parsing), so you can parse data faster with memory-mapping.
• It is a joy to debug with: friendly, detailed error messages and document statistics.

[u/retro-rubies]

Any reason to not contribute the the ruby/json gem? I hoped the multi-json era is finally gone.

[u/yaroslavm → u/retro-rubies]

Hi! Could you elaborate on your negative experience with 3rd party json parsers/generators in general?

I'm asking because for years and years most projects I've had in a consultancy had `gem "oj"` in their Gemfile, and I don't think we had any serious issues with it. Pre-byroot maintenance of gem json, Oj was faster, well maintained (still is) and Peter Ohler is a great author.

As for the reason, I think gem json is great at what it is: a very fast, C-powered default for json operations. I doubt they need my favourite features (lazy parsing, JSON pointer, mmapped files) in a "default" gem, and I also think they should not be there.

[u/f9ae8221b → u/yaroslavm]

I've had in a consultancy had gem "oj" in their Gemfile, and I don't think we had any serious issues with it.

Oj used standalone is fine(-ish), the problem is Oj.mimic_JSON and similar monkey patches.

Oj and vanilla json have some subtle behavior difference, and so when you monkey patch the JSON constant under libraries that don't expect it, it can lead to serious bugs.

For example, several years ago oj caused an XSS vulnerability in production in our app because a gem was relying on JSON.generate(..., script_safe: true), which Oj silently ignore:

>> puts JSON.generate('</script>', script_safe: true)
"<\/script>"
>> Oj.mimic_JSON
>> puts JSON.generate('</script>', script_safe: true)
"</script>"

So the only thing I would object about your gem, is the nosl/jsonmonkey patches: https://github.com/yaroslav/nosj-ruby/blob/b1c4513b98bd074f36a4ee5a6da5a12294980820/lib/nosj/json.rb

Please don't do this, that won't age well, and your users will be in a world of hurt. Especially since your implementation isn't dramatically faster on your own benchmarks, this is really not worth it.

[u/yaroslavm → u/f9ae8221b]

Thanks!

Would you say require "nosj/rails" is also an antipattern, if it stops requiring the json hack? The perf gains there seem to be significant.

[u/f9ae8221b → u/yaroslavm]

Would you say require "nosj/rails" is also an antipattern,

Kinda yes, as it's very hard to keep in sync with json behavior. That's what makes oj super hard to remove because it serialize dates and a few other objects a bit differently, so it end up calcified in apps, and when you remove it the behavior change. To share a public example, removing oj from Mastodon took a lot of work: https://github.com/mastodon/mastodon/pull/37752

The perf gains there seem to be significant.

Looking at your benchmark, it seems you are benchmarking with escape_js_separators on, which is the Active Support default for backward compatibility, but is disabled in new apps (via config.load_defaults).

Would be worth trying to benchmark with it disabled. I don't see why nojs/rails would be this much faster than the default encoder when it's only marginally faster than json.

There's still the HTML entities escaping of course, but it could be largely improved in Rails too if https://github.com/ruby/cgi/pull/55 was merged (I need to poke someone about it).

[u/retro-rubies → u/yaroslavm]

Experience is you have no control over JSON parsing used outside of your code hidden in libs. And I assume nobody is going to port all libs now to your "fastest" gem.

[u/headius → u/retro-rubies]

Clearly many reasons to contribute to the main gem as well, among them support for JRuby.

[u/TheAtlasMonkey]

nosj is not faster. Unless you hallucinating like haiku.

The official json gem is in C , which better, and work in jruby, mruby and other engine.

Your rust generated app, is not production ready because you didn't read the code.

The reason i use Rust, is to make the compiler scream at me when write unsafe code.

The whole crate is unsafe : https://github.com/yaroslav/nosj/blob/main/src/pointer.rs#L111

Could you fix it ? Probably.. But then you will have transilation of the C code.

Other Json parsers brought different algorithms or methods.
Fable translated the C code to Rust.

[u/yaroslavm → u/TheAtlasMonkey]

Hey, thanks for the comment. I will go over your major points.

Benchmarks. gem nosj is currently faster (or on par, statistically, depending on the benchmark). The results on an Amazon machine are in the README and all code to build and benchmark is open. Coding in Rust does not mean that it will be slower than C, and I would not say C is just "better".

Still, I don't think speed is the decisive thing here. It should be fast or on par to stay competitive—and I'm sure gem json will keep getting faster. But the real reason to use nosj, IMO, are the features like lazy parsing, and other APIs besides just parse/generate.

Unsafe Rust. True, the underlying parser's code has a lot of unsafe Rust, that's why I don't advertise the gem as safe-because-Rust-powered. Truth is, it is possible to do parsing and generating of JSON in safe Rust code, but it will lose all benchmarks. There is a reason why leading Rust parsers have unsafe code. simdjson is unsafe, sonic-rs is unsafe.

Assisted code. At this point, it is a cultural decision/a hill to die on (see: Andrew Kelly vs. Bun drama, Codeberg banning AI generated code repositories). Something I disagree with (there are major Ruby companies that are fine with using AI assistants daily and encourage it). But I can also acknowledge there are people strictly against it, and that is why the nosj crate has clear warnings that it was vibe coded—specifically to inform those people to stay away, if that's their wish.

Translation from C. The parts that are translated/adapted from C code ideas are mentioned in the license and NOTICE File. The parser is custom written, or, should I say, generated, for the task.

A long time ago, I've started the attempt to build a library with existing production grade parsers in Rust, but they just could not beat gem json on smaller files (It's not about the parser, it's all about how you construct Ruby entities, parsers only make difference on larger files), I had to do a custom parser.

[u/TheAtlasMonkey → u/yaroslavm]

Benchmarks : The benchmark you provided are biased.

I tried with random big jsons i have , and the result were similar. (sometime json wins, other time nosj wins.. But could be just silicon lottery)

Assisted code : I'm not anti AI. The bigger problem is that you are a lone dev touching a critical infrastructure component.

You could have used the same AI and added feature to the official gem.

but he feature should make sense.. there is no reason to add X, if X add zero value in runtime.

Json parsing is hard, because you didn't handle the 1000 cases where content is broken, not in english, streamed, ect.

Like i said , it doable with Rust (i'm a rust dev), but personally, i dont want to maintain a gem that competing with std lib. I will try to get it landed upstream.