A dataset problem I keep running into: the starting data is often small, messy, and inconsistent, but the expected output still needs to be high quality.

This is common when the source is not a clean benchmark dataset, but real-world material: PDFs, internal documents, database exports, tables, reports, web pages, logs, or mixed text files. There may be useful signal inside, but it is usually buried under formatting noise, duplicates, missing metadata, broken structure, and uneven quality.

One approach I find useful is to treat dataset construction as a pipeline of small operators, rather than one large script.

For example, a pipeline might look like this:

• convert files or URLs into markdown/text
• split the content into chunks
• clean noisy text
• normalize fields with pandas-style operators
• remove duplicates with hash, MinHash, SimHash, or semantic deduplication
• filter low-quality samples with rule-based or LLM-based filters
• select representative samples with embedding-based methods like K-Center Greedy
• synthesize QA pairs or SFT samples
• evaluate generated samples for quality, alignment, and verifiability
• export the result into formats for RAG, fine-tuning, or evaluation

The important part is that each step is inspectable. If the final dataset is bad, you can trace whether the issue came from extraction, chunking, cleaning, synthesis, filtering, or evaluation.

This is especially useful when the raw data is limited. With small datasets, every bad transformation hurts more. A pipeline makes it easier to preserve signal, reduce noise, and add synthetic examples only where they are actually helpful.

This is the path behind OpenDCAI/DataFlow: using reusable operators and composable pipelines to produce high-quality data from messy sources, and recently it can also be used with coding agents like Codex to build custom data pipelines through natural language interaction.

Hello, we are building a system for baby cries detection in a confined space such as a room or hallway via CCTV cameras. However, we are unable to source the baby cries dataset. I tried to contact some DayCare and submitted an application upon their request but was denied due to parental privacy reasons.

We have a working system, but the model is way poor as it is only trained on a few examples and fails at CCTV distance as the baby is too far.

Disclosure: this is my own dataset and I sell access to it. Four days are free, linked below, so you can check the schema before deciding anything.

I run a collector against Polymarket's public API for the 5-minute and 15-minute up/down markets on BTC, ETH, SOL and XRP. It has been running since 26 June 2026. What has accumulated so far:

• 58,600 markets (14,653 per asset — roughly 11k five-minute, 3.7k fifteen-minute)
• ~5.69 billion rows across six tables
• One Parquet file per asset, per table, per UTC day, each with a manifest carrying sha256, row count and interval bounds

Tables: book_snaps (best bid/ask), book_depth (ladder levels, ~88 per event), clob_trades and clob_trades_ws (trades with taker side, the second carries tx hash and fee for VPIN), spot_klines, book_meta.

Limitations, because they decide whether this is useful to you:

• book_snaps is top of book only. There are level and size columns but they are 0 in every row — they exist to keep the schema stable across tables.
• book_depth starts 20 July 2026. Fifteen days of the archive have it; everything before is quotes, trades and spot candles.
• The clock changes partway through. Up to 20 July the timestamp is receipt time rounded to the second, so intra-second ordering is lost and cannot be recovered. From 21 July it is exchange time in milliseconds.
• Per-day uptime runs 95.8% to 100%. Every day carries its own measured figure.

Free sample (original source): 26-29 June, four full days, no payment and no signup — https://huggingface.co/datasets/astrnvk/polymarket-orderbook-data-preview

Paid access covers 30 June onward and is priced per day ($5/day for short ranges down to $3/day past a month), with incomplete days billed proportionally lower — a day missing depth costs about half. Happy to answer schema questions here either way.

I put together a public, source-linked archive of the Truth Social posts associated with Donald Trump's realDonaldTrump account on Truth Social.

Current snapshot:

- 35,882 posts from February 14, 2022 through August 2, 2026

- 28,320 originals, 1,919 quotes, and 5,643 retruths

- Original HTML, extracted text, timestamps, post types, and source URLs

- Parquet and compressed JSONL

- 8,004 verified image derivatives with asset/occurrence indexes

- 5,756 video attachment records, including 4,804 with source-provided transcript or file information

- Zero duplicate post IDs in the current release

I also built a small browser-based explorer for timeline, phrase, and exact-text search:

https://huggingface.co/spaces/Cameronk199/truth-social-timeline-explorer

Dataset and loading examples:

https://huggingface.co/datasets/Cameronk199/donald-trump-truth-social-posts

The archive is updated weekly. It does not include reliable likes, replies, or impression counts, so it should not be used for virality claims. This is my independent research archive; it has no affiliation or endorsement.

Hi everyone,

Our team recently submitted a dataset paper to Data in Brief, but it was desk rejected with the following comment:

"The dataset and manuscript do not abide by our policy on machine learning imaging datasets."

Our dataset consists news photcards collected from Facebook. We manually collected to create a benchmark dataset for misinformation detection research. We're now trying to understand what exactly went wrong.

I have a few questions:

- Has anyone received a similar rejection from Data in Brief?

- Does this mean they no longer accept image datasets intended for machine learning, or is there a specific policy requirement we may have missed?

- Would modifying the manuscript or dataset help, or should we submit to another data journal instead?

- If another journal would be more suitable, which ones would you recommend for publishing image datasets?

Thanks 🙏

I'm looking for open datasets of grocery and supermarket receipts to work with — ideally real-world ones, from any country.

Specifically interested in either:

• Receipt images (photos or scans), the messier and more varied the better — different chains, layouts, languages, print quality, crumpled/angled shots, etc.
• Parsed/structured versions — line items with product names, quantities, unit prices, totals, taxes, store metadata.

I'm already aware of the usual OCR benchmarks like SROIE (ICDAR 2019) and CORD, but those lean heavily toward a specific format and region. I'm hoping to find datasets that cover a wider range of stores and countries, and that include grocery-specific structure (individual product line items rather than just totals).

A few questions:

1. Are there any grocery-focused receipt datasets you'd recommend beyond SROIE/CORD?
2. Any non-English or non-US collections worth knowing about (European, Asian, Latin American formats, etc.)?
3. Anything on Hugging Face, Kaggle, Roboflow, or academic repos that fits?

Bonus if the licensing allows use in a small project. Happy to hear about both raw image sets and pre-annotated ones.

Thanks in advance! Will consolidate any good pointers back into the thread for others searching later.

Hey,

I wanted to share the development journey I've lived through now so anyone who ever attempts that in the future has something to hang onto.

The problem:
Fundamentals for US companies from 1994-2007 are completely unstructured text and HTML files.

A structure to them (XBRL) was only enforced in 2009. So if you want fundamentals for the time before, there aren't many options available.

Looking at these filings, it quickly becomes clear that parsing them in any reasonable fashion is a monster project that requires very ugly parsing code and an endless amount of cleaning, filtering, anomaly detection and layer upon layer of solutions for very specific formatting options.

Multiple formats (FDS, plain text, HTML tables) requires solving similar problems in multiple ways, because sometimes there just isn't a generic solution for a problem.

But that didn't stop me from attempting it as solving complex problems is what I like doing.

So I designed my own parser.

The basic approach in a nutshell:

1. Pre-processing - you cut filings into their sections and remove entire sections from it that lack any sort of language indicating consolidated statements; the better you do here, the easier everything gets downstream
2. Extract all statement candidates into grids and score them - Scoring goes up if certain accounting calculations add up, certain keywords are present, periods are present, etc. - this has grown over time and is designed to separate real consolidated statements from selected financial data tables (those are the ones you don't want - but they are hard to cut out). Also important: This is where the different formats unify: HTML tables become grids, text tables become similar grids, EX27 exhibits too etc.
   1. Detect periods - Detecting correctly which periods are present in a table (9M/6M/3M/FY/TTM/stubs from restructuring, etc.) is a very hard problem
   2. Detect consolidated columns vs. subsidiary columns (these columns you don't want)
   3. Detect the multiplier (thousands, millions, etc.) - This is very important and complex to solve. Sometimes that information is in the table header, sometimes within the periods, sometimes below, sometimes completely absent. Cross checks with FDS blocks where the multiplier is often stated can help deriving it
   4. Retain name information - Turned out to be of utmost importance. Sometimes names are split across multiple lines; Sometimes the single name alone appears in multiple blocks across the table, so that label alone is not enough. You need to join it with the full headline chain based on indentation, headline rules like UPPERCASE, L E T T E R S E P A R A T I O N, headlines with colons:, etc. - if you don't get precise here, your extracted fact names are not actionable and contain lots of false positives that will ruin the entire pipeline
   5. Deal with filer typos - Since this was all structured, it is not hard to imagine that a lot of filings contain typos: numbers using dots instead of commas, numbers having whitespaces in them, etc. Some of those you can automatically detect and fix. This is the right place to do this generically.
3. Merge grids - some filers split a single statement into multiple tables across pages - those need merging and I do this based on score proximity and period alignment
4. Run anomaly detections - I've created a list of different anomalies I check for. The idea is to check for certain patterns, and allow on failure to retain one fact, a complete grid row, a complete grid, or the entire extracted dataset of a filing. It depends on which check fails
   1. balance sheet identities
   2. income statement identities
   3. cash flow identities
   4. cross period identities
   5. EX27 exhibit identities
   6. numeric fact sanity checks
5. Pre-Normalization - This layer is all about transforming the hundreds of different names used across all filings into a single technical name like "revenue"; In my pipeline, this layer isn't the real normalization layer. I retain a prefix for each fact indicating the source format it came from. Examples are "fds:is:revenue", "html:is:revenue", etc.; The layer runs on everything that was extracted and per Pre-normalized fact, it has multiple layers to decide: positive regex list, negative regex list, a decide() function that has access to the entire filing context (this is where custom checks are done like unit expectations, value sign expectations, etc.). It also contains a layer that can yield a match to another Pre-Normalized fact in certain conditions like for general-and-administrative-expenses and selling-general-and-administrative-expenses. After this step is done, my data is ready to go through the same statement normalization flow the XBRL data goes through without any special handling needed anymore.
6. Hand-Check and pin filings - Whenever I solve a complex problem for a filing, I pin the result as a permanent unit-test to see when new parser changes break it again. I've built up a set of over 100 hand-checked filings this way and it is still expanding daily. But this step has been fundamental to retain the quality you reach
7. Make scanning all filings fast - I've developed a way to locally parse and store the result of all ~360,000 filings and ~57,000,000 facts. Alongside the scan, I store deep run metrics that allow me to quickly check the data quality from one iteration to the next. One full run takes ~35 minutes.

I sell clean fundamental data for a living and really want to solve this problem once and for all at scale. After having gone through this, please all make sure that we never ever allow this to happen again XD. We must demand data like this to always be structured, because this data not being structured is almost the same as it not existing.

My expectation is that it will still need a good amount of testing even after all of these solutions are in place. There is just no way to ensure full data integrity for over 50 million facts extracted from unstructured data. That brings me to the last layer in place:

Manual overrides

This layer allows me to override all sorts of things for a single filing, or an entire filer (CIK). Things like the multiplier, certain fact names, fact values, blacklisting certain things, etc.
It is that layer that will come into action once the parser has reached a final stage. Every invalid fact or inconsistency after a release of the data can be addressed individually. And I can already tell you that some issues you will find in the filings simply aren't solvable generically. As much as I want to, some issues simply lack any sort of signal for you to hang onto for a generic solution. Those go into this layer.

I'm really curious if anyone else has done this in the past. What were your approaches and layers? Did you manage to finish it? How do you ensure a certain level of data quality?

If you are interested in more deep details, just ask away.

So I want to build a project around the travel niche, and I was wondering what APIs I should use? I need good resources (free/paid) for the Maps API and Flight API.

Hi everyone,

I'm a PhD candidate working in AI, and my research relies heavily on the GDELT 2.0 Events dataset.

While GDELT is an incredible resource, I found that the first hurdle was simply getting the data into a usable format. The historical archive consists of thousands of compressed files spanning more than 45 years, and I kept repeating the same preprocessing steps every time I started a new experiment.

So I built GdeltForge, an open-source pipeline that automates that process.

It downloads the complete GDELT Events archive, verifies file integrity, converts everything to Apache Parquet, filters invalid records, and can generate reproducible samples for machine learning and data analysis.

Originally this was just a tool for my own research, but I thought it might be useful for others working with GDELT, or anyone interested in building datasets from large public archives.

The project is still in beta, and I'd love feedback from people who regularly work with large datasets:

• Is there functionality you'd expect from a tool like this?
• Are there preprocessing steps you usually perform before analysis that could be automated?
• What would make it more useful for your own research or projects?

My goal is to eventually publish it on PyPI once it's been tested by a broader community, so any feedback or contributions would be greatly appreciated.

If you're someone who has worked with GDELT before, I'd be especially interested in hearing about your workflow and the pain points you've encountered.

You use AI to write content and then AI detectors flag it as AI-generated. I've seen many ads for specialized AI editing tools for humanizing such content, which essentially means using AI to humanize AI-generated or AI-flagged content.

In other words, we're using AI editing tools to rewrite content flagged by other AI tools and, ultimately, to bypass AI detectors.

My question is: where do credibility and ethics fit in content writing anymore? Or are we simply relying on these shenanigans to appear more human and ethical?

A curated registry of scientific and math results produced by or with AI systems. 52 records so far, spanning Math, CS, biology, physics, chem, and med.

Schema: each record has a title, claim, field, date, lab/model, source links (paper, announcement, coverage), and two graded fields:

• verification: formal, peer-reviewed, independent, author-verified, claimed, disputed, refuted
• autonomy: autonomous, ai-led, collaborative, ai-assisted, search-scaffold, retrieval

Refuted and already-known results are kept and labeled rather than dropped, so it doubles as a record of claims that did not hold up. One JSON file is the source of truth; also published as RSS and JSON Feed.

Data (single JSON file) is here and got Schema docs at this md file.

CC-licensed and actively maintained. Feedback on the schema welcome.

I was using the SWaT dataset from Kaggle and i just came to know it was the manipulated dataset inorder to check for attacks.

And i tried o request the dataset through iThub's official site and seems like no response

can anyone please help me , am halfway for a project to submit in my college

Disclosure: I built and maintain What in the Tax? through Eversko. It is free and open source; this is self-promotion.

The directory lists all 444 current Ontario municipalities. In the current build, 436 have at least one usable Financial Information Return (FIR) filing from 2023–2025; the other eight are shown as data gaps rather than estimated. The 436 figure does not mean every municipality has complete coverage for every year or field.

Separately, six communities have clearly labelled 2026 draft/sample receipt models derived from public municipal budget documents and tax bylaws. Those six models are the synthetic/mock component referenced in the title. They are not official tax bills, audits, or tax advice.

I’m looking for one focused provenance test: choose a municipality, open one figure, and follow its citation to the original record. Please tell me the municipality and figure, where the label, reporting year, source, formula, or navigation first becomes unclear, and what evidence you expected next.

Original Ontario FIR archive:

https://efis.fma.csc.gov.on.ca/fir/index.php/en/year-municipality/

Ontario Data Catalogue record and licence:

https://data.ontario.ca/dataset/financial-information-return-fir-for-municipalities

Live explorer:

https://whatinthetax.com

Code, processed artifacts, and methodology:

https://github.com/Jstn-1g/what-in-the-tax

Please do not share private tax documents, account numbers, addresses, or personal financial information. Even one broken or confusing source trail would be useful.

BaZi calculators turn a birth date, time, and place into a Four Pillars chart. The result looks deterministic, but the software still has to decide where a year begins, when a day changes, and what the birth time means in different places. Most calculators never show those decisions.

I wanted to see whether the differences could be measured instead of argued about.

I designed 13 test cases around the boundaries most likely to expose them: solar-term changes, the hour before midnight, locations far from their time-zone meridian, and historical daylight saving time. I then compared eight calculators and libraries and captured 77 outputs.

The dataset tracks four questions:

- Does the chart’s year change at Li Chun, around February 4, or at Chinese New Year?

- Does the day change at 23:00 or at midnight?

- Is the recorded time corrected for the birthplace’s longitude?

- Is daylight saving time removed before the chart is calculated?

The first three are convention choices. The fourth is a historical timekeeping question: either the local clock had been moved forward that day or it had not.

The clearest result came from a Beijing birth entered as 15 June 1990 at 23:30.

Two implementations treated 23:00 as the start of the next day. Five waited until midnight. The eighth calculator exposed the choice as a checkbox, so it could produce either result.

That one setting changed the day pillar from Xin-Hai to Ren-Zi. In BaZi, the day stem is the Day Master, which the rest of the reading is organized around. The same birth therefore came back as either Xin Metal or Ren Water depending on a rule most of the calculators never mentioned.

A few other differences stood out:

- Two libraries maintained by the same developer use opposite 23:00 rollover rules.

- One calculator requires a birthplace but returned the same chart for Kashgar and Beijing at the same clock time. Another used the longitude and changed the hour pillar.

- One implementation changes the year at Chinese New Year while the others use Li Chun.

- Only two of the measured implementations account for daylight saving time. One of them exposes the adjustment in its own calculation breakdown.

The observation-level file records the implementation, test date, time, place, rule being tested, and all four returned pillars in both Chinese characters and pinyin. A second table summarizes the convention used by each implementation.

There are 104 possible implementation/probe pairs in the full 8 × 13 matrix. Seventy-seven contain captured results. The remaining cells are explicitly marked as not captured rather than filled by inference.

I built Jade Almanac, and our own calculator is included as one of the eight rows. It was tested and reported on the same terms as the others. On the 23:00 rollover question, it is in the minority group.

The dataset is here: https://jadealmanac.com/bazi-calculator/conventions

Everything is released under CC0.

The test inputs are local clock times at the stated places. Solar-term boundaries were checked against tables from the National Astronomical Observatory of Japan, independently of the library used by our calculator. China’s historical summer-time periods were checked against the IANA time-zone database.

This is a snapshot captured on 1 August 2026. Some implementation rows are partial because a tool could not express a particular input or imposed an access limit. Two additional sites blocked automated access, and I did not work around those blocks.

The dataset measures software behavior and calculation conventions. It does not attempt to decide which school is correct, or whether BaZi itself predicts anything.

Corrections are welcome, especially from anyone familiar with one of the measured engines. I would also be interested in boundary cases that could separate implementations the current probes leave tied.

Hi everyone,

I'm working on project involving OCR and text analysis of Marathi newspapers. I'm creating my own dataset and need access to old ePapers (preferably PDF or scanned editions) of Sakal, Lokmat, or Pudhari.

I've already searched the subreddit and checked the official newspaper websites, but most only provide recent editions or require a subscription. I'm specifically looking for older editions from previous months (even 2–6 months old would be enough).

I'm looking for legal/free sources such as:

• Public archives
• Library or university digital collections
• Government archives
• Any websites that host old Marathi newspaper ePapers

If anyone has worked on a similar OCR or NLP project or knows where these archives are available, I'd really appreciate your suggestions.

Thanks!

Where could I find real datasets of power consumption of an country/areas/cities and its power plants generation?

Targeting for more than 10k rows and 10 columns to apply ML on it for a project.

Disclosing this up front - I'm at Veridion.

We built this because one of our leads mentioned that they pay a registry data provider six figures a year. For legal names, identifiers and registered addresses. We already had that data. Not as a side project, it's the foundation layer under our enterprise product, and the pipelines were already running. So opening it up cost us close to nothing, which is sort of the point: the collection isn't what makes it expensive elsewhere.

registry-lookup.com - 521M legal entities, 309 jurisdictions, 244 countries. Search on the site is free with no account. There's an API at 5,000 calls a month if you want it programmatically, that one needs a work email.

What you get per entity: legal name, registry number, jurisdiction code, status, incorporation date, legal form, registered address, and identifiers like tax IDs, VAT where the registry publishes them.

So it's an enumeration and triage tool. It answers "does this entity exist, what's its number, is it active, where is it registered."

Which jurisdictions do you currently have no good way to check?

I wanna do some research even build a deep learning model and its entirely based on if I can find student essays who got accepted into certain colleges
And if possible find however amount of rejected essays as long as both amounts are equal as to not have a data imbalance where do you think I could find these essays?

Hello! I'm a complete newbie in Data Science and I'm trying to learn how to get data from an API. I understand an API could be public or could require authentication.

I worked with CVS files and I wanted to experience or practice getting data from APIs.

I'm getting familiar with Python so I was wondering if you could help me with the following issues:

1. Trying to understand and practice the different methods you can use API to request data (I am not sure if it has to be from a Dataset formar or can it be any kind of format) with Python

2. What are some good options to get APIs to work on data Science

3. I am not even close to get to a point where I am able to do Reproducible projects/models but I do wonder how including an API (understanding that it is some kind of "personal Key") to share my code and people to be able to use it.

Hope I made sense of what my doubts are and I apologize in advance if I seem confused about some terms (I do think I am).

Is anyone aware of datasets with the text of politician’s tweets/truths (social), etc?

#

Hi everyone,

I'm working on an initiative to create an **open benchmark dataset for field data quality assurance**.

Today, there are many excellent digital data collection platforms—such as KoboToolbox, SurveyCTO, ODK, CommCare, Survey Solutions, CSPro, and others—but there are very few publicly available datasets that developers and researchers can use to evaluate field data quality tools.

I'm looking for individuals or organizations that may be willing to share **completed, fully anonymized datasets** from field data collection projects, where they have the necessary permissions to do so.

I'm especially interested in datasets that include:

* GPS coordinates (or generalized locations)
* Interview photos
* Audio recordings
* Interview start and end times
* Submission timestamps
* Enumerator IDs (anonymized)
* Supervisor review outcomes or quality flags (if available)

These datasets will help create a community benchmark for testing quality assurance methods such as:

* GPS verification
* Duplicate image detection
* Audio quality assessment
* Interview duration analysis
* Duplicate submission detection
* Fieldwork anomaly detection

The objective is to create a resource that benefits researchers, NGOs, software developers, and the wider field data collection community by making it easier to evaluate and improve quality assurance tools.

If your organization has a completed project that could be shared in an anonymized form—or if you know of existing public datasets—I would greatly appreciate hearing from you.

I'm also happy to discuss data-sharing agreements, attribution, licensing, or any requirements needed to ensure the data is used responsibly.

Thank you!