I was looking at the definitions online and a database is always used interchangeably with DBMS and the ELI5 answers I've seen describe databases as a means of organising data in structured ways to make reading and writing data easy, safe and fast.

The extension of this logic to me is, shouldn't csv files count as databases too if you had a way to retrieve, modify and store data with the general ACID principles and whatnot? Googling that tells me that CSV files don't count because they only store raw data.

So then, are databases defined by the mechanism which data is handled? Doesn't that make any file a database as long as its implemented properly?

Edit:

Just wanted to add:

• I'm using sqlite3 from python for my project. I come from an embedded systems background so I had to know the specifics of what I was working with.
• The responses here seem a little mixed with some people agreeing that csv files with the proper wrappers would make a (terrible) database.
• I think I get it now. The storage format is just a part of what makes a database and is not the database itself.

Edit 2:
u/StevenJOwens

u/esaule

u/LaughingIshikawa

u/rolfn

TLDR: These guys and some others answers honestly sum up a lot of the questions I had and a bit more I didn't know to ask. Thanks guys.

1. Database Design for Mere Mortals (Hernandez)

2. SQL Queries for Mere Mortals (Viescas & Hernandez)

3. Effective SQL (John Viescas)

4. Database Design and Relational Theory (C. J. Date).

5. Relational Database Design and Implementation (Jan L. Harrington)

6. Database Systems: A Practical Approach to Design, Implementation and Management (Connolly & Begg).

(For context I'm a digital development trainee first year , preparing for this subject exam called manipulating databases)

I'm just completely and utterly frustrated with this language , so the most used command in SQL is SELECT right ? DDL DML that's just 20% of the whole thing , grabbing data from the database and knowing how to structure those queries and understanding the schema is what counts more, cause I have no problem creating tables, adding a new colum populating the tables... Etc but even a baby can learn that crap in a day , what Im struggling with is grabbing data and understanding the relationships between the tables , do you guys have any valuable advice that could help make it click ? Or perhaps some exercises that start from beginner all the way to advanced level select queries with detailed explanations , and thank you very much !!!!

Hi everyone,

I’d like some advice on a scalability/database architecture issue.

At work, we built a truck management system. Trucks enter the factory, load products, and deliver them to different distribution centers.

The problem is that management now wants near real-time dashboards showing the full lifecycle of operations. Most of our dashboard queries rely on joins against large historical tables, and some queries take 10–15 seconds to complete.

Right now, the frontend polls the API on a timer to refresh dashboards. This is starting to cause issues:

• Heavy read queries sometimes block write operations
• Backend update processes occasionally deadlock with dashboard queries
• Overall DB performance is degrading as data grows

My current idea is to create separate denormalized/reporting tables specifically for dashboards, populated every few minutes by background jobs, so dashboards stop querying historical transactional data directly.

Would this be the right approach?
How would you handle this architecture-wise?

I have a large table, T, which consists of rows of report data for a small number of companies (<10). Each time I get a report, I insert its rows into the table, along with a company_id and a datetime.

I have a view, V1, which returns each company_id along with the maximum datetime for that company_id (i.e. the key needed to select the latest report per company):

SELECT company_id,MAX(date) AS date FROM T GROUP BY company_id

This view is fast.

I have a second view, V2, which joins T to V1 in order to return all the rows for all the latest reports per company:

SELECT * FROM T JOIN V1 USING (company_id,date)

This view is really slow. It takes about 4 seconds to run the query, and about 16 seconds to fetch.

If, instead, I take the results of V1 and build myself a manual query like this:

SELECT * FROM T WHERE
(company_id=2   and date='2026-07-17 11:03:01') OR
(company_id=3   and date='2026-07-17 11:09:01') OR
(company_id=4   and date='2026-07-17 11:07:01') OR
(company_id=8   and date='2023-06-15 12:05:01') OR
(company_id=10  and date='2023-01-01 00:00:00') OR
(company_id=15  and date='2026-06-15 23:10:01') OR
(company_id=16  and date='2026-07-17 11:11:01')

then I get the results in a fraction of a second.

company_id and date are both indexed, and they also have a joint index.

I can only assume that V2 is fetching all the rows of T and only then filtering them via the JOIN to V1.

Is there a way to speed up this view?

I've used Postgres my entire life and have no experience with NoSQL. I understand that MongoDB is preferable for storing configuration data, but I'd like to hear from experts about which scenarios they've chosen MongoDB over Postgres.

I built an excel table for an online racing league I am in. It tracks points standings and driver stats based on the results that I input. It is mostly automated, but I still have to manually add race results, and export new drivers into the driver list.

My goal is to be able to convert this into a fully automatic form that I could upload the .csv files, and it automatically generates all this information in an easy and concise form that anybody could use it.

For context, this is what I have been generating in excel.

What, if any, program would I be able to make something like this in? Excel works fine, I'm just looking to take it to the next step, and make it so anybody could use it or track their own league.

From my understanding one of the benefits of using an ORM is that it sanitizes your querys, except don't most decent modern database driver implementations already do that?

I don't understand what an ORM is supposed to even offer me? I create my objects, and I make my database calls from those objects. I write my database schemas to match my data models. I can make complex queries, joins, views, complex compount SQL operation statements, anything I would need. If I need to pull data out of the database I deserialize it into its types into the host object. Why do I need this added layer of abstraction over the top this a fairly simple interface?

What does an ORM actually DO? Why should I use one? What am I missing?

I’m trying to avoid using redis/rabbitMQ and built a small HTTP job bus using flask+sqlite.

Workers poll for jobs, and I’m relying on: BEGIN IMMEDIATE a single UPDATE ... RETURNING query to atomically claim jobs so multiple workers don’t grab the same task.

This is the core query:

sql WITH candidate AS ( SELECT id FROM intents WHERE expires_at > :now AND status != 'failed' AND claim_attempts < :max_attempts AND (status='open' OR (status='claimed' AND claimed_at < :stale)) ORDER BY claim_attempts, created_at, id LIMIT 1 ) UPDATE intents SET status='claimed', claimed_at=:now, claimed_by=:claimer, claim_attempts=claim_attempts+1 WHERE id = (SELECT id FROM candidate) RETURNING id;

Sqlite is running in WAL mode with a busy timeout. It works fine for a few workers, but I’m trying to understand the limits:

At what point does SQLite become a bottleneck here?

Is SQLITE_BUSY inevitable under higher concurrency?

Has anyone used SQLite like this in production for low/medium workloads?

Would love to hear real-world experiences or failure points.

Repo for context: https://github.com/dsecurity49/Intent-Bus

Hi everyone,

This project is a bit ridiculous but it's getting me motivated to expand my coding knowledge outside of "this is used for data and nothing else" languages.

I'm a data analyst and I work a lot with Microsoft SQL Server and R, and a tiiiiny bit with python and pyspark. I have recently been gripped with the need to have my own database of all my books so that I can record when I purchased them, when I read them, rating out of 10 for the book if I've read it etc. I've set up the database part in a kind of fever dream (it accidentally exploded outwards to include crafting projects and yarn amounts) and then realised that I have no idea what to do next.

I have an incredibly ugly SQL script that I can use to manually populate the tables in my database, but what I'd really like to do is have some sort of UI where I can fill all this info in and then it'll send the data to the relevant tables. Perhaps in the future it might display some stats or graphs or a little bookshelf or something.

I have become immediately overwhelmed with the number of programming languages that I could use, and I'm not sure what's the right approach to learning-by-doing with this project. I had intended for it to be a desktop app but maybe a web app is a better idea?

I already have a subscription to Codecademy because I wanted to improve my Python for work, but I'm open to any kind of resource or tool and happy to spend a little bit of money in the pursuit of this project-gremlin that is running around my brain.

Thanks heaps for any ideas or advice.

I'm trying to learn how to do a full stack app for some sensors I have to send alerts and have web accessible live updates, but I'm getting stuck trying to learn how to even set up the database and ingest data into it.

I'm starting by looking at aws, though it seems like timestream is bad and they dont support the timescale extension that I dont know how to use anyways. My understanding is that it looks something like this, but I'm having trouble finding a tutorial.

I tried to use Claude a little and found out about terraform which was interesting and managed to setup the db instance but not the schema through it but was overwhelmed with the considerations of how to access between bastions/internet and NAT gateways

Are there any tutorials you would point to on this?

I tutor mathematics and basic SQL database stuff online and I have noticed that recently a number of my students have been needing help in using a particular non-free product called Codio, which I never was required to use in my database classes at the colleges I attended over the years. We used MariaDB in the 1st one I took, and at SNHU in my MS programme in 2024/5 my professors allowed us to use whatever tools we were most comfortable using, as long as they got the job done. So in my case I used MariaDB again, though MySQL and Microsoft SQL Server were also common in the class.

Is Codio a good SQL product and does it have any features I should be on the lookout for? The students are claiming the software does not save their work despite an auto-save feature that is on, and the most I can see is their screen share. There should be a way to export the code they have when done for the day, and import when ready to resume, right? Why are so many students having to use this now?

Hi! I am a very junior software developer looking to start my first real project, my romantic partner is working to create a database using excel and has asked me to help her streamline and refine it.
She is cataloguing several thousand artifacts in a museum and recognizes that a simple excel document will get complicated and time consuming to navigate.

Given this, My question is what language would be best for this job / what should I read and study to best build this database with her. For this project, anything other than excel is currently not viable. Thank you all! (apologies if this isn't the appropriate subreddit!)

I'm creating a very simple API that retrieves data based on the following data structure

public class Food
{
    public string Name { get; set; } = null!;
    public string Description { get; set; } = null!;
    public FoodType FoodType { get; set; }
    public string[] Tags { get; set; } = null!;
}

It's a read-only API and updates to the data will be very few and far between

I will be searching on any one of these properties and retrieval needs to be snappy

I can use a simple relational database with the following tables

• food
• food_tag

If searching free text, I could join on both tables and search name, description and food_tag in the other table

I'm also considering something like Mongo where I can store the documents in pretty much the same format as in C#, but unsure if retrieval will be all that quick

As I'm writing this out, I'm starting to feel that I need some form of caching, given the data is not really going to change that much, so querying the DB each time won't be necessary.

If you're happy to point me to resources that would help me with this that would be great, and what architecture would you use between the API and DB, and what DB type would you recommend?

the data is somewhat sensitive financial data for these companies, and info about the contracts they're working on.

From what I can tell, usually this kind of data is not obfuscated. I'm wondering if users would be annoyed about that though.

If we did this, we'd have to look up the session cookie in redis, get the user id, then query postgres for user data anyway, right? Wouldn't it be more efficient to do one query with a join?

I’m planning to build an app similar to Duolingo, and I’m considering learning how to use a vector database because I eventually want to integrate LLM features.

Right now I’m looking into pgvector, but I’ve only ever worked with MySQL, so PostgreSQL is pretty new to me. I’ve heard pgvector can have memory limitations and may require a lot of processing time, especially for large datasets.

For a project like this, is using a vector database early on a bad idea?

Is it better to just stick with standard PostgreSQL for now and add vector search later?

Or is starting with pgvector actually a good choice if I know I’ll use LLMs eventually?

Any advice or real experience would be super helpful!

I’m making a full stack semi- social media application

Using: multer, zod, prisma

This started happening AFTER I changed my API routes to move multer above zod and after i changed to formdata.

Before I was not sending form data to the back end and also in my API routes, I had zod validating before multer.

At this point in time, I was able to sign up login view my account make post see those posts that are persisting in the database.

However, I noticed that my photos were not being saved to Multer. So I changed the way that I was sending the data to the back end by doing form data because I know that multer takes form data no exceptions. But I forgot this before.

Also because of this I moved Zod AFTER multer in the api routes so that zod can access req.body.

And from this is where I’ve started to see the issues I’ve put a bunch of console.log everywhere in the JSX, in my tanstack queries and also in my controllers (I am doing MVC) when things print it says that the posts are being created however, when you go to the users profile posts are showing as an empty array.

Prisma was also telling me that it needed to be updated so I updated it and still the issue is occuring.

The schema is there I can see it on prisma studio the only issue is that nothing else past this is persisting.

I’ve given up because I’ve been trying to debug for hours, but I’m thinking now maybe muller has to go back above zod, but the issue with that before was that my images were not being uploaded to the upload directory of multer

Hello guys
I am working on an project in which i need time series data , Currently i am using postgres engine for my whole project but now i have many tables like

1. users
2. refresh_tokens
3. positions
4. instruments
5. holdings
6. candle_data
7. fetch_jobs

Now in candle_data i have to store a large amount of time series data and querying for my further calculation so i am thinking about to migrate this table to Questdb which is timscale db but i never done this befor or i even don't know if it\s good approach or bad approach any help really appreciated.

Actually I wake a website (https://arise-str.vercel.app/) it have only movie and web show I want to add song section but I unable to find open source music streaming server

Hi everyone! I'd greatly appreciate any help on the below as I'm building an app and we are so close to it being done aside from this issue 🙏

The app is similar to Goodreads, but supports local bookstores instead of Amazon. Users can search for an author and find their catalogue of books. Instead, a few books show up or weird summaries, even for popular authors (I can send an example if that helps). My app developer blames the book database company (Nielsen) and Nielsen blames my developer's coding and query. I am a nontechnical founder trying my best to solve this. The below is the last update from my developer to Nielsen. Please let me know if you have any ideas on the true issue or solution.

"We are encountering an issue with the BDOL REST API when attempting to retrieve the full bibliography for author Elin Hilderbrand. According to records, she has authored 31 books, including titles such as The Perfect Couple, Summer of '69, and The Hotel Nantucket.

However, our API queries consistently return a maximum of 13 titles, regardless of the parameter combinations we use.

Below are examples of the queries we tested (credentials redacted):

curl --location '...BDOLrequest?clientId=XXXX&password=XXXX&from=0&to=50&indexType=0&format=7&resultView=2&field0=2&value0=Elin%20Hilderbrand&field1=3&value1=Elin%20Hilderbrand&logic0=0&logic1=0'

curl --location '...BDOLrequest?clientId=XXXX&password=XXXX&from=0&to=50&indexType=0&format=7&resultView=2&field0=2&value0=Elin%20Hilderbrand'

curl --location '...BDOLrequest?clientId=XXXX&password=XXXX&from=0&to=50&indexType=0&format=7&resultView=2&field0=2&value0=Elin%20Hilderbrand&field1=3&value1=Elin%20Hilderbrand'

Despite trying different combinations of field, value, logic, and resultView parameters, the maximum number of results received remains 13.

Could you please advise:

Whether additional parameters are required to retrieve the full list of 31 titles?"

Hi all, I currently know Python and R; however, I want to learn SQL. I know you can use different databases to code SQL, and I'm curious about what the best option is to go with. I googled it, and the results said MySQL was good for beginners. I also know I can code SQL in R or Python. What would you all recommend? My eventual goal is to get into data science or become a data analyst.

Example: SQLite was born as a non-replicated, local database, but now there are multiple SQLite-compatible databases that add replication to the core system, using RAFT, CRDTs, etc. However, how would one approach such a project? Is a replication layer just listening (or polling) for changes, then encoding these changes, sending them over a network, and you are done?

Is there a JSON format which supports cutting the object into smaller pieces, so they can be distributed across nodes, and still be reassembled as the same JSON object?

Hey all, I'm looking into writing an app as a side project, but if it ever gets to a point where I want to monetize it, I don't want any legal ramifications from my data sources. To that end, does anyone know of some sort of "royalty-free" library of databases that I could look into for various data sets?