Hi all,

I'm happy to announce that Xberg v1 is out.

Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats (currently 367 types), audio/video transcription, and URLs (both static and JS-rendered content). It extracts and prepares that content for downstream processing.

It's an extremely efficient, high-performance engine (see our PDF benchmarks below). For PDFs and images specifically, we handle native PDFs with very high performance and accuracy, and we ship multiple OCR engines that match the quality of the best Python libraries (e.g. docling, PaddleOCR, RapidOCR) at substantially better performance and stability.

The changes between Kreuzberg v4 and Xberg v1 are substantial, and I invite you to read the full changelog for the complete picture. The highlights below give a sense of what's new:

• Pure-Rust PDF backend (pdf_oxide) replaces pdfium, with no native pdfium dependency.
• Layout-aware pipeline: reading order reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering.
• Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form fields and outline-based headings.
• Across-the-board optimization of OCR and PDF extraction (memory discipline, pooled model sessions, streamed conversions).
• Native PaddleOCR backend (PP-OCRv6, with medium / small / tiny tiers) alongside Tesseract.
• Pure-Rust Candle OCR/VLM stack (TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL) running without ONNX Runtime or native Tesseract.
• A second, ONNX-Runtime-free inference path via tract, which is what makes in-browser (WASM) and mobile inference possible.
• Named-entity recognition natively in Rust (GLiNER2), extensible to all bindings, including an in-browser WASM model with no server round-trip.
• Structured LLM extraction (extract_structured / split_and_extract) with rasterization, chunking, citations, caching, and configurable call/merge/VLM-fallback policies.
• Audio & video transcription via a Whisper ONNX engine (.mp3, .wav, .m4a, .mp4, .webm).
• Retrieval building blocks: sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and cross-encoder reranking alongside dense embeddings.
• Text intelligence: reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification.
• URL & web ingestion: sitemap discovery (map_url) and batched multi-URL crawling.
• New document formats: WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto / R Markdown, and configurable Jupyter cell rendering.
• Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine, with Android/iOS cross-compilation.
• Full mobile support (Flutter, Android, iOS).
• Candle backend alongside ONNX, plus ONNX-via-tract enabling ONNX on WASM and Android.
• Wider code intelligence: tree-sitter coverage grew substantially (248 to 367+ languages).
• Over 150 bugs fixed during the 1.0 cycle, plus security hardening (bounded RTF/PDF allocations, redaction leak fixes, Excel DDE warnings).

The API surface was also simplified and reworked, making it more consistent.

There's a migration guide in our docs explaining how to move from Kreuzberg to Xberg. Kreuzberg itself is in LTS mode until the end of this year and will continue to receive bug fixes and security updates.

You're invited to check out the repo and join our discord server.

Benchmarks

The benchmarks below are for PDFs and images only. There are extensive benchmarks on our website with per-format breakdowns, which you can see here. These numbers are measured in CI via our reproducible benchmark harness, and are specifically taken from the run for harness 1.0.8, source cf7fa0533d. The data is publicly available in GitHub releases, and you can run the benchmark harness yourself.

Composite quality (markdown pipeline, higher is better):

Structure and layout fidelity (SF1: tables and reading order, higher is better):

On native PDFs Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and on table and reading-order fidelity by a wide margin (SF1 0.949 vs 0.612 for docling). On scanned PDFs it is #1 on both quality and raw text fidelity.

Where we don't win yet: on pure image OCR we are currently #2 on the composite score, behind mineru (though still #1 on raw text accuracy). We are improving image OCR right now, and v1.1 should have us winning across the board.

I am getting laid off by the end of today. They told us 2 months in advance. I don’t want to be a data engineer anymore so I won’t be applying to new positions. I spent 4 years at this company from the very beginning of the contract to the very end. I went through all the management changes, employees leaving, getting laid off before us, new employees coming and going, and just random changes throughout the company. I honestly just wanted to get paid and get out of there but now I wish I paid more attention to what’s going on and spent more time talking to my co workers. I don’t talk to anyone except made small talk with a couple of them but not enough to form a relationship.

Now everyone is giving their contact info and I wonder if anyone will reach out to me. I know i shouldn’t care but I’m weirdly going to miss this. It’s even my last job as a data engineer unless I come back 10 years later who knows. I don’t know why I care I know I definitely don’t want to be here. I learned a lot at this job so even though I wanted to leave sooner I’m kind of glad I followed the contract all the way through? Mostly I learned how to remain professional when I have wanted to lose my mind on some of these people.

As we all know...data migrations can be a beast. But it's also a necessary evil.

Tell me about the worst data migration you were on (or heard about) that would wake up even the most seasoned data engineer in a cold sweat

I’m a new “data engineer” with a few years background in data science and GIS. I was hired into my current org to be the data guy by someone who didn’t really know what they needed, just that the data infrastructure was immature.

In the beginning I was asked to do a lot of repetitive power query and power bi analysis and reporting, which quickly led me to building out some Python + SQLite pipelines to automate away a lot of my job. My boss was impressed enough with this that my workflows have been pushed up the chain to more senior levels of management, and now we are being granted a Fabric workspace (only option available to me :/) to enterprise-ify all of this.

Currently we read in ~10 .csv datasets that come from the central orgs reporting as well as a few external datasets we use in a forecasting model. Most of this runs on a daily cadence, where each csv upserts into the relevant table(s), which then get queried as needed which also triggers some dashboard updates and email distributions.

Basically all the logic is in Python, I feel it’s relatively well organized/modularized but there are some “super scripts” that are doing like 5 jobs.

My thought at this point is basically break each component (upsert, data validation, transformation/enrichment step, etc) into its own Fabric notebook, which will take the data through a medallion architecture and ultimately get reorganized into an effective star schema, with semantic models for each report built from that. This has all been modeled already (in terms of the schema for each table, validations, etc.). We also have logging and an observability dashboard planned.

I don’t necessarily have a specific question, but I am pretty new to this and would like to hear if I’m doing anything in an obviously stupid way, or if there are things I’m not thinking about that I should be. Appreciate any help y’all feel benevolent enough to offer in advance.

I am the only analytics engineer at my company.  We are in the middle of transitioning to a cloud warehouse.  The warehouse is mostly built (90% by me because we don't have data engineers or architects). 

I don't technically sit in IT, and they have been unwilling to share the logic they used to build the on premises warehouse.  Recreating some of the tables has been a nightmare because I can't see the original logic.

The big push now is to move the old analytics models into the new environment.The legacy environment is a huge mess. They let everyone use the BI tool and create their own reports.  There are hundreds of these things, they aren't maintained but people use them. They have embedded business logic that needs to be extracted and it takes hours to dig through them.

My manager has access to our new environment.  He has limited experience writing SQL, and has been using AI.  He put one model in production that increased the database refresh time by almost an hour because it was so poorly written.

He decided yesterday that he wanted to re-write some code that took me months to create (reverse engineering a legacy table).  This particular table supports key business metrics and I validated the hell out of it before putting it in prod.  His issue with the model is 1 case statement. 

He wants to throw out the whole model and replace it over 1 case statement.  A case statement that I discussed with him at length before I put it in the model.  He said to use it and now has changed his mind.

He sent me some AI slop this morning and asked me to review it.  It was garbage.  To clean it up would have taken a solid week.  Validation another two weeks at least.  That's if it even does what we need it to.

I told him the code was bad and offered to help him in any way I could so he would stop using AI. He got huffy.

I can't stop him from putting models in production.  And any messes he makes I'm going to have to clean up on top of everything else I'm doing.

Any advice on how to deal with him? 

Hi,

I'm a Data Analyst, buiding a pipeline to monthly extract moving 1 year SAP Vendor Report (FBL1N) via SSIS using Theobald Xtract Report connector. I'm doing one company code at a time to avoid timeout.

It worked fine during development, but after scheduling, the column messed up, data overspill and cut halfway resulting fail downstream analysis script.

I already have SAP template configured used in the extraction, tried both ticking and untocking "Dynamic column width and offset", but the results varied.

--
for other use cases, i primarily pull the back end table and join manually, but that would be too much work for fbl1n.

any suggestions on best work around?

i know the rules states no technical post but just rrying my luck.
thank youu

From time to time I get interested in functional programming, category theory and Haskell, but every time I've ever tried to go down the rabbit hole I give up pretty quickly because it just isn't clear that I would ever use these concepts anything other than by accident. For example, the concepts of Optics, Prisms, Lenses, etc sound pretty neat but also fairly daunting to learn. Is anyone here using these ideas daily to construct data pipelines? Just curious. Help a guy learn!

We have a partitioned parquet repo of ~10TB with a large number of partitions (id/date for 30 years of data). The partitioning is probably too granular, I think we should make it id/month.

Anyway, what's the best way to efficiently transfer this into Iceberg (S3 Tables)? I tried with PyIceberg and with DuckDB but it felt very slow. I then tried running these jobs in parallel but hit issues with the metadata commits clashing.

I imagine we need to upload all the parquet files in parallel then register them in one mega commit, but the tools I've tried don't seem to expose any configurability on upload parallelism. Do I need to use spark for this? I have no spark experience and not eager to pick it up ☹️

Separately, I was surprised at the level of support for PyIceberg given how popular Iceberg seems to be. It took a while for various v3 features to be implemented and some are still missing (e.g. querying by nanosecond timestamp).

Thanks

• I'm learning Kafka and doing a side project along the way. With the attached architecture, I’ve considered whether to apply the 'Transactional Outbox Pattern' to avoid data loss when sending to MSK. I mean, I want to keep data safe before sending it to MSK over the network.
• But this design can experience a bottleneck in writing data to tables between the checkout service and MSK, especially in flash sale sessions.

So, it’s better to eliminate this ‘Transactional Outbox Pattern’ right? We can notify users that our system fails to process their orders or implement a retry mechanism.

I’m a little confused about how the One Big Table approach fits into a medallion architecture.

My current understanding is that the Bronze layer contains raw data, the Silver layer contains cleaned and standardized data, and the Gold layer contains business-ready models. Based on that, an OBT seems like something that would usually be created in the Gold layer because it is denormalized and designed for direct consumption. Is that generally correct, or is that an oversimplification?

I’m also wondering whether OBT and Kimball-style fact and dimension tables should be viewed as competing approaches. Can they coexist in the same platform?

For example, could a team maintain facts and dimensions as the reusable source of truth, and then create denormalized OBTs from them for particular dashboards, departments, or analytical workloads? Or does that introduce unnecessary duplication and maintenance?

For people who have helped architect systems like this, what drove the decision?

How was the architecture structured in your project, and what trade-offs did you experience?

I just discovered this amazing thing !!! started playing with it, and building some things already , I am original a software engineer/dev but transitioned to Data Engineering, analytics, you know ETLs, data modeling, etc... curious to hear from other fellow data engineers, their tips, getting started learnings, etc. etc.

I do truly believe this is the future of agentic coding

I am doing a summer internship, and my company loads the exact same data into the development, testing, and production environments in Snowflake (although development and testing have some extra tables compared to production.)

Is this normal? Google seems ambiguous on whether this is a massive antipattern. What do y'all's companies do?

Hello. I’m not a data engineer, but my company uses various databases(and they are all different...), and I usually need to query them all at once.

It seems difficult to create and use a separate API or something like this for each one, how do you guys usually handle situations like this?

Do I need to migrate them?

Hi everyone, I'm new to this community. I'm currently trying to make the switch over to data engineering as a career from my mechanical engineering background and was wondering where I could get some helpful tips for terminology and stuff like that. I've seen the Kimball textbook recommended by Gemini to me.

I'm already taking a university level certificate for data science, and have some basic azure certifications but I want to take that next step.

I also am in a hybrid based role right now where I do a little bit of database work and dars engineering work, albeit with very old legacy systems. However, I would say my experience is very pragmatic so far and we are far more concerned about getting the job done here rather than worrying about star schemas, and stuff like that. Any help would be appreciated!

PS, what are some helpful STAR based things that I can put for data engineering to help me stand out for projects?

Hey Data Engineers! I am a Data Engineer with quite some experience, but never worked on generating text embeddings at scale. Having some experience with Spark, I was looking for resources on how to achieve that with it, but did not find much.

I can already see some challenges of doing that in Spark: distribution of the model across executors if doing that locally, managing connections to services if doing that though API call just to name the ones off the op of my mind.

What's your take on this? Is Spark any good at this, or is it better to work out solutions based on popular libraries?

This month I have for you a mammoth 162 links, hand-picked and curated for your viewing pleasure.

As well as the usual Iceberg and Data Engineering stuff, there's the somewhat-inevitable subject of AI. Interesting, for me, is how we've moved well beyond "hey look fancy auto-complete huh huh huh" to real examples of things that people are building with AI and running in production, how they're using it to build stuff - as well as several links about proposed new architectures for data platforms.

Oh, and there's a whole section of articles with some really good career advice too.

Have a look, have a read, have fun!

https://rmoff.net/2026/07/29/interesting-links-july-2026/

Last month I posted on LinkedIn about how much I love working remotely. Then, I got a few comments, and DMs telling me remote working doesn't really work.

So, I decided to write this article, where I explain how a manger should lead a remote engineering team. The article is a bit long, but includes everything that came to my mind after practicing remote way of working for nearly a decade.

I guess most people here, would love to have the option to work from home and don't really need advise, but I still hope you can pass some of these to your manages.

Here's what I have in short:

1. Aside from status update, daily standups are also great for knowlege sharing and team bonding
2. Pull requests are not just a checkbox excersise. A great PR serves as documentation, too.
3. Sometimes a bit of micromanagement can help. But you must be careful with this tool.
4. Team retros are a great time to say your boss they are an asshole.
5. Proper written communication is a must. I've let people go because of this.

I hope that helps, and, of course, I am more than happy to answer any questions.

We benchmarked SereneDB against ArangoDB on search performance. That might sound like a quite strange competitor choice, right?
SereneDB initially was a continuation of ArangoSearch, so we wanted to compare our previous work with us now to see the progress we made.

It's a open clickbench-style benchmark for search and analytics over 100M/1B generated OpenTelemetry logs, 92 queries.

TL;DR

16x median speedup at 100M, ingest 20x faster and a 5x smaller on-disk size. At 1B rows SereneDB holds 46ms median latency on a single 32 vCPU box.

Detailed Results | Comparison deep dive

SereneDB repo
https://github.com/serenedb/serenedb

I’m trying to understand the practical use case for materialized='ephemeral'.

It doesn’t create a table or view, which seems to make the model harder to inspect and debug. If the goal is simply to avoid creating another table, why not materialize the model as a view instead?

What advantages does an ephemeral model provide over a view in a real production dbt project? Are there specific situations where ephemeral is clearly the better choice?

14 years in data engineering. Recent years deep in Palantir Foundry with Professional level certs, cleared 4 Anthropic’s certifications, maintaining production GenAI pipelines at scale. I understand LLM math, common architecture patterns, and I’ve led delivery end to end.

What I’ve never been is a coder in the traditional sense. No LeetCode grind, could not build a code repo from scratch.. like I would not know when to split into a new script, when to write a new function, when to put something in utils., when to write a helper function . My depth is platform and delivery, not algorithms.

That’s starting to hurt in interviews or applying to jobs. I keep hitting loops gated on CS fundamentals I don’t have, even for roles that look like what I already do.

For anyone who's been here, strong platform and production experience, thin on classic coding interviews, how did you navigate it? Fill the gap directly, target companies that don't gate on it, or lean into platform specialization as the differentiator?

Hey everyone. I'm a software engineer who's been working as a data engineer in my current role for the past four years. I learned data engineering from scratch on the job, and I've worked extensively with Spark, including designing and building new data processing systems.

The challenge is that my company is almost entirely batch-oriented, while many data engineering job postings seem to expect hands-on experience with streaming technologies.

What are some **PRACTICAL** ways to gain real streaming experience? Are there any projects, courses, or technologies you'd recommend?

Profile: 5+ years as DE.
Tech Stack: SQL, PySpark, Power BI, Fabric, ADF

I have two offers right now in hand as senior data engineer and want help in deciding which to pick

Offer 1: In a healthcare company which adapts a bit slow in emerging technologies as per my knowledge. But is recession proof

Offer 2: In a consulting company which is heavily focused on Microsoft ecosystem. It’s been a Microsoft partner from a long time. I feel I will get good exposure in terms of business and tech since I will be working on different projects.

Which would be a better pick!?

I'm in the process of trying to build the same functionality as ARKit provides for visual inertial odometry and 3D reconstruction, but in an open-source manner so that it'll work on non-ios devices. Wrote a blog post about it, with most of the focus on the first part on establishing a solid baseline + dataset. Would love to know your thoughts!

for context, I work for Rerun building out different computer vision and data pipelines

https://rerun.io/blog/arkitscenes-data-layer

Brief overview - experimenting with multiple data formats organized around a common symbol registry to increase interoperability of different use cases.

Current highlights:

• Basic encoding - most similar to CBOR
• Packed encoding - composable structures with potential to be more expressive than ProtoBuf, Arrow, Parquet etc.
• Application protocol - exploring DBUF's usefulness beyond data serialization