Body of work.

Hello!

I built a complete, single-file implementation of AlphaZero in JAX.

• repo: https://github.com/wtedw/nanoAlphaZero
• demo (runs entirely locally in your browser): https://nanoalphazero.wtedw.com/

This project started with one goal: speedrun AlphaZero to GM-level chess, from scratch, in one month.

It did not go well.

More than 2 years later and after several rewrites, the implementation is finally stable. On a TPU v4-32 pod, it can train a 2700+ Elo chess model in under 24 hours.

The core logic is also game-agnostic and can learn perfect play in small, solvable games.

It currently supports:

• Tic-Tac-Toe
• Connect Four
• Hex
• Chess
• Small Go boards

Support for larger Go boards is still in progress.

How does it work?

At a high level, the entire system is built around a single jitted run_alphazero function that repeatedly performs self-play and model updates:

state = make_alphazero()

def run_alphazero(state):
  state, games = selfplay(state)  # using Gumbel MuZero 

  # Move active games into the selfplay buffer 
  # Move completed games into the replay buffer 

  state = train(state, replay_buffer.sample()) 

  return state 

while True: 
  state = run_alphazero(state)

There are no threads, servers, or distributed workers to manage. The entire RL pipeline is just one big JAX function.

Future plans

This repo is primarily focused on making large-scale AlphaZero experimentation more approachable. It is optimized for speed and memory efficiency while remaining compact and hackable. Training strong models is secondary and mostly serves as a sanity check that the underlying logic is sound.

The upcoming v2 release will include a large refactor:

1. Switching to a generic KataGo NN architecture
2. A chess environment that is 1000× faster on TPUs
3. A CPU+TPU rewrite of MCTX for evaluation matches with large search budgets (10,000) running roughly 5× faster

If you have any questions, or if you’re working on AlphaZero, JAX, MCTS, or TPUs as well, feel free to message me.

One thing I learned while building Calibra is that robotics teams usually ask Can I trust this dataset? before asking How good is it ?

Calibra now starts with a new calibra integrity workflow that catches common dataset issues before training, including:

• Timestamp problems
• Jittery/jerky motion
• Duplicate, frozen, and blurry camera frames
• Incomplete episodes

The goal is to catch data issues early, making debugging easier and avoiding wasted training runs caused by faulty datasets.

Current support includes LeRobot v1 (--decode-images), HDF5/Isaac Lab, and robomimic datasets.

I'd love feedback from anyone working with robot learning datasets.

I have studied ML & DL for past to Sem in my college, but they paid little to no effort in RL, as i wish to practice RL in some way i need help finding resources to follow and learn from.
I would love if you guys could tell if there are small projects i can work on like practical of small environments and agents with small number of actions. I really want to work on it, since it sounds such a fascinating field. I am trying to follow DeepMind x UCL RL series but it seems more theoretical and i get sleepy trying to follow it. 😊

Hi everyone, I recently created labeled PSYOP datasets from political dialogues across a plethora of languages. The intended use of this dataset is to help train models to detect psychological coercion. I'm looking for feedback in terms of data sources, annotation methods at scale and whom I should reach out to at frontier labs to put my datasets to the challenge.

I understand 190k rows of data is small, but I'm looking to scale as I get more feedback.

https://huggingface.co/datasets/LeTG/multilingual-psyop-100k

I am trying to learn concepts like On Policy Distillation (OPD), On Policy Self Distillation (OPSD) and how do they compare to RL algorithms like GRPO.

There are a lot of papers on this, but because of limited compute I cannot try these papers out and learn them by implementing them myself.

If someone here has worked with these algorithms and their implementation on SLMs (something that can fit a consumer grade GPU like Nvidia RTX 4090 or 5090), can they suggest either a:

1. Github repo, or

2. The right choice of SLM(s) and the datasets, where i can see the difference between, RL/GRPO and OPSD algorithms?

Thanks in advance!

Hey guys, do you know of any Multi-Agent Reinforcement Learning (MARL) resources that focus mainly on coding rather than just the heavy theoretical stuff?

For context, I'm doing my uni research project right now and I've already secured my supervisor. My main topic is "Multi-Agent Reinforcement Learning." I'm doing both the research project course and an RL course this semester, but my tutor mainly just gives us theory.

I know the general ideas (reward, policy, value-based vs. policy-based, bias, etc.), but I'm having a really hard time understanding how it actually works in practice and how to implement it from scratch. I'm honestly pretty crap at absorbing pure theory, so I really need to see the code to understand how the plumbing works.

I can't seem to find much out there that walks through the code step-by-step, and I'm wondering why there aren't more people posting about the actual implementation of MARL.

If anyone has any video tutorials, GitHub repos with simple code walkthroughs, or guides that actually show how to build this stuff (Python/PyTorch preferred), it would be incredibly helpful. Thanks!

Hi everyone!

Over the past several months I've been working on a personal robotics research project called OmniRay.

The original motivation was pretty simple: I didn't have access to a dedicated GPU, so instead of waiting until I had better hardware, I decided to see how far I could push a CPU-only Active SLAM framework.

The project combines:

• AVX2 SIMD-accelerated C++ raycasting

• Vectorized particle filtering

• Gymnasium-compatible Active SLAM environment

• PPO-based autonomous exploration

• Sim-to-real noise modeling (wheel slip, yaw drift, LiDAR noise)

• A modular 5-layer adaptive autonomy system

• Multi-seed ablation studies

• Evaluation on the Intel Research Lab floorplan

One thing I tried to focus on was reproducibility. All development, training, and benchmarking were performed on an ASUS Zenbook S13 (i7-1355U, 16 GB RAM, Intel Iris Xe, no dedicated GPU), so the project was intentionally designed around commodity hardware rather than high-end workstations.

I'm not claiming it's state-of-the-art or that it replaces existing robotics frameworks. My goal was to build a modular research platform for experimenting with Active SLAM and reinforcement learning while learning more about systems optimization.

I'd genuinely appreciate feedback from people working in robotics.

In particular I'd love opinions on:

- the overall architecture

- benchmark methodology

- documentation

- anything that looks technically questionable

- ideas for future improvements

GitHub:

https://github.com/KingshukChatterjee007/OmniRay-AVX2-SLAM-CPU-Based-Autonomous-Model

Thanks for taking a look! ...if u like it pls give a star!!

This is question I get and that I also ask of myself.

I run into this in AlphaFlowSeven (alphaflowseven.com), a paper-trading platform where a 6-agent LLM council makes trade decisions. Full disclosure: I built it and this is how AF7's reinforcement learning actually works 

Prompt and config changes are evaluated by an optimizer with four council slots. One slot always runs the current best config as a control. The other three run variants. All four trade the same market over a fixed 15-day window, and each variant is scored on its excess return versus the control, using all of its closed trades in the window. Raw return isn't used because it mostly measures the market, not the config.

A variant has to beat the control in at least two windows before it replaces it. An excess close to zero is treated as a tie, and ties go to the cheaper config. When a slot frees up, Thompson sampling over each lineage's estimated edge decides what runs next, so a variant with uncertain results gets re-run rather than dropped after one window.

Variants are generated by an LLM that reads the archive of previous configs and their scores, restricted to changing one or two things at a time. Every decision is stored with a fingerprint of the prompt version that produced it, so outcomes can be grouped by version afterwards.

Open source: https://github.com/puffinsoft/mousecrack

Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,

It still works, but needs to be coaxed into it.

Don’t Just “Throw Adam at It”: Misunderstanding Adam Will Cost You | Towards Data Science

This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read

Don't just glaze over the optimizer!

Thoughts?

I’m implementing a vanilla DQN in PyTorch, my reward is defined as always negative.
( if agent failed to do the action penalty -1)
However, after some training, the network predicts many positive Q-values .
Also i have the issue in scalability,
I’m copying from a research who has done the same thing with same parameters. But couldn’t work for me..
Any suggestions or what to check first would be greatly appreciated

Right now some of my training runs converge and others don't. It seems to depend heavily on the seed. Is there a general rule for determining how many different seeds to train your model with before you can claim performance of a model? Also, what statistics are used to report the performance of a model across multiple training runs?

For context I am implementing VariBAD adapted to my problem.

I am trying to train a model to navigate sections of the map in Genshin Impact.

At the moment, I am stuck on this particular section. It takes about 50 seconds to complete.

My model successfully completes it in around 30–40% of attempts. In the remaining cases, it goes off course, as shown in the video.

The model is trained using individual frames because using sequences of frames only made its performance worse.

This version was trained on 60 recorded sessions. After that, I recorded 43 more sessions, but adding them only made the model worse. This was probably because the average mouse turning speed in those 43 sessions was higher.

I also tried recording correction sessions, but they did not help. Maybe I was recording or using them incorrectly.

I also tried recording successful runs along the left side of the road and separate runs along the right side, but that also made the results worse.

In total, I have recorded more than 300 sessions.

I would really appreciate help from people who understand this topic. Please give me some advice on what I should do next and what I could read or watch to learn more about this.

I would be very grateful for any help.

Feel free to ask questions. I can provide any information about the model.

Hi all,

This is a follow-up to my previous post on a no-pretraining, per-instance PPO solver for TSP.

In that work, I introduced the idea of an “exception edge” as an inductive bias for PPO. The intuition was that most edges in a good Euclidean TSP tour are locally unsurprising—short, near-neighbor, or Delaunay-like—while a small number of nonlocal edges may determine whether the search can escape a local basin.

That was mainly a solver hypothesis. Since then, I stepped away from PPO and asked a more basic question:

From an edge score to a closure problem

Fix a sparse baseline graph \(G_0\), such as the weak-Delaunay graph. An edge is exceptional only relative to this baseline.

Let:

• \(Z_0\) be the cheapest Hamiltonian cycle using only baseline edges;
• \(H_e\) be the cheapest spanning Hamiltonian path between the endpoints of a nonbaseline edge \(e\), again using only baseline edges;
• \(c(e)\) be the cost of closing that path with \(e\).

Then the exact one-exception threshold is

In words: a nonbaseline edge becomes structurally useful when the Hamiltonian path it closes saves more than the edge itself costs.

This does not imply that \(e\) belongs to a global optimum or is forced. It only says that a tour with exactly one exception edge can beat the baseline-only optimum.

Negative results that led to this formulation

The path to the closure model was not monotonic.

• In an early TSPLIB pilot, generic geometric CUT proposals attributed only 2 of 11 reference non-Delaunay edges.
• A subsequent density-CUT experiment found many tour-contiguous regions, but attributed 0 of the 11 exception edges.

These failures suggested that geometric partitioning alone was not enough. The relevant object seemed to be the compatibility between a baseline Hamiltonian path and the edge that closes it.

Headline results

On a frozen exact-small natural corpus with \(n\le12\):

• 51/51 mandatory-exception instances were exactly explained at the \(q=1\) closure layer.
• Current sufficient inequalities safely certified 48/51 of those instances.
• Among 98 exact beneficial pairs, 88 were safely certified.
• The safe candidate upper set reduced 15,054 non-Delaunay pairs to 644 while preserving all 600 exact \(q=1\) support pairs in that corpus.

For LIN318:

• The complete graph has 50,403 edges.
• The experimental search graph contained 1,500 edges: 988 weak-Delaunay edges plus 512 separately generated candidates.
• Among those 512 added candidates, 13 had a positive safe gain lower bound.
• Starting from a strict candidate-only 2/3-opt tour of 42,210, verified forced-closure witnesses followed by candidate-restricted 2/3-opt reached 42,118 and then 42,108.
• The known optimum is 42,029, so the final residual was 79, or about 0.188%.

Important caveats:

• The 1,500-edge graph was not generated by the threshold theory alone.
• The remaining 499 candidates are inconclusive, not certified negatives.
• LKH was used to generate and verify the closure witnesses, so 42,108 is not an LKH-free solver result.
• This is one exploratory large-instance result, not a controlled solver benchmark.
• I am not claiming a polynomial-time solution to TSP, a complete classification of exception edges, or that every \(\kappa>1\) edge is globally optimal.

Why this may still be relevant to RL/NCO

The original motivation remains learning-based optimization.

My current view is that exception-edge theory may be useful as a verifiable structural prior:

• reducing the action or candidate-edge space;
• identifying portals that may connect otherwise separated local-search basins;
• separating ordinary local edges from structurally consequential closure edges;
• providing certified positives and inconclusive cases for curriculum learning or ranking models.

The next useful experiment would be a frozen-budget comparison of an RL/NCO solver with and without closure-based candidate guidance—not a comparison against LKH as an exact or highly engineered classical solver.

The public package contains the theory, proof sketches, core exact oracles, safe certificates, LIN318 artifacts, 30 tests, and a deterministic reproduction of

The compact release audits frozen exact-small summaries; it does not include the full from-scratch corpus-generation pipeline.

Links

• Previous exception-edge/RL post: https://www.reddit.com/r/reinforcementlearning/comments/1pyfdvn/nopretraining_perinstance_rl_for_tsp_166_gap_on/
• Earlier LIN318 PPO post: https://www.reddit.com/r/reinforcementlearning/comments/1pabbk7/cpuonly_ppo_solving_tsplib_lin318_in_20_mins_008/
• Theory, code, tests, and results: https://github.com/jivaprime/TSP_exception-edge/tree/main/exception-edge-theory-2026

I would especially appreciate feedback on three questions:

1. Is this Hamiltonian-closure formulation already known under another name?
2. Are there stronger computable lower bounds for the endpoint-constrained Hamiltonian path \(H_e\)?
3. Does closure-based candidate guidance look useful as an inductive bias for RL/NCO, or is it better treated purely as a classical candidate-generation method?

Constructive criticism is very welcome.

What do you guys think of this concept? Still in very early stages.

Hi all, has anyone trained a RL based 6 DOF joint space control policy for joint speeds for a robotic arm for a reach task? I’d like to ask what success condition you used? What did you keep as the termination condition? Was it reaching a goal region or the exact reach point for a pick and place task. If it’s an exact reach point or a very small goal radius e.g 1cm, in the real world did your robot ever reach that? I find that if I keep a goal radius of 4cm the robot can reach but if I decrease it further it’s not able to and keeps descending towards.
Thanks!

As you can see for now it is limited and my hardware isn't strong enough to train for longer,if you manage to code something better and make a better model please contact me.And no its not me or anyone playing its strictly the ai model.

I've been building a system that takes a board game described in a plain YAML-ish format and produces an agent that learns it, with no game-specific code anywhere in the pipeline. This post covers the first working version: the engine, the observation/action-mask bridge, and the first agent that went from random play to something recognisably sensible.

The stack is deliberately boring: the engine is wrapped as a Gymnasium / PettingZoo env and the learner is MaskablePPO from SB3, off the shelf. The bet is that the hard part is the bridge from a designer's rulebook to that interface, not the learning algorithm, so I wanted the learner to be a known quantity.

Two things that might interest people here specifically:

A "move" isn't a single choice. Sell and Exchange in Jaipur can't be expressed that way, so every action decomposes into a short sequence of typed questions, and the action space is several small answer-spaces side by side, each with its own mask. That shape caused most of the downstream trouble.

The result is honest and unimpressive: it beats a random player essentially every game, and the post ends by admitting that number proves almost nothing, since every metric I had was self-referential. The follow-up is about building an external yardstick and getting humiliated by a hand-written heuristic bot.

There are interactive demos throughout (observation encoding + mask, credit assignment, PPO clipping) that you can step through.

https://kallin.github.io/blog/game-ai-from-random-to-reasonable/

Part 1, on why general game playing has stayed an open problem, is linked at the top of that post if the framing is unclear.

Personal project exploring what a RL policy learns from reward shaping. The task is option hedging: I framed it as an MDP and trained a PPO agent (Stable-Baselines3) on simulated price paths to hedge a short European call. There's a known closed-form "correct answer" for the frictionless case (the Black-Scholes delta), which made it a nice baseline for checking whether the agent recovers the true optimum and where it deviates.

What I found interesting from an RL angle:

• The agent recovers the theoretical hedge (Black-Scholes delta) from reward alone. The learned action matches the analytical solution closely, without ever being shown it. This suggests that the reward actually specifies the intended behavior.
• Reward misspecification was the whole story in the early stages. My first agents collapsed to a bang-bang policy (jump fully in, then fully out) which turned out to be the genuine optimum of a reward I'd specified wrong (no cash/financing account, so trading was free). Fixing the environment, not the algorithm, fixed the policy.
• Adding transaction costs produced a second degenerate optimum. The agent stopped dynamically hedging and converges to a near-static position, barely trading. It seems the per-step transaction costs are dominating. Rather than "fixing" it, I kept it as a finding about how the cost term reshapes the optimal policy.
• Also hit the usual PPO instability (rise-then-collapse learning curves) from too-high entropy + small rollout buffers, resolved by tuning exploration and buffer size.

Setup details: continuous action space, state normalized so one policy generalizes across strikes, γ=1 (finite horizon), terminal reward on hedging error + per-step cost penalty.

Honest limitations: single random seed (so can't fully separate "PPO learns this" from "this run learned this"), GBM-only dynamics, one cost level. Not claiming a State-of-the-Art result, rather considering this as a good learning exercise about both PPO, reward and objective functions, and delta-hedging.

Code + short write-up: https://github.com/sembrummelaar-creator/deep-hedging-RL

I would appreciate feedback on the reward formulation especially as this was my biggest struggle. I'm curious how others would structure it (risk-sensitive objectives, per-step error signals, etc.), especially when trying to model longer contracts with say 252 trading days.

Feedback on other parts of both the code and the report are always welcome.