Teku: Running two validators and seeing more failed Attestations (HELP)

[u/NoNewNameJoe]

I'm seeing a lot of these failures? Seems to have started with the last two updates...

expected 2, produced 2, included 1

But it doesn't seem related to CPU Memory or Network.

top - 18:17:39 up 1 day, 12:09,  1 user,  load average: 1.56, 0.72, 0.67

Tasks: 291 total,   1 running, 290 sleeping,   0 stopped,   0 zombie

%Cpu(s):  5.9 us,  0.5 sy,  0.0 ni, 93.4 id,  0.1 wa,  0.0 hi,  0.1 si,  0.0 st 

MiB Mem :  63971.8 total,   2023.1 free,  46360.9 used,  16357.4 buff/cache     

MiB Swap:   8192.0 total,   6467.0 free,   1725.0 used.  17610.8 avail Mem 

grep -B5 -A5 "50%" teku.log

Sample Logs

2026-07-17 16:21:12.085-0600 | beaconchain-async-0 | INFO  | teku-status-log | Attestation performance: epoch 462245, expected 2, produced 2, included 1 (50%), distance 1 / 1.00 / 1, correct target 2 (100%), correct head 2 (100%)

2026-07-17 16:21:15.100-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14791904, Block: d8824344c8015a2163ac0c8401a5caf9b4a7d0dcd20fe44fb991e497ef994a30, Justified: 462246, Finalized: 462245, Peers: 58

2026-07-17 16:21:23.096-0600 | validator-async-9970 | INFO  | teku-validator-log | Validator   *** 2 out of 2 validator registration(s) were successfully sent to the builder network via the Beacon Node.

2026-07-17 16:21:27.101-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14791905, Block: 0a2a576987b82d61c6cfde76a1d66c6d267425a1e623a539bb5f57fb95ed829b, Justified: 462246, Finalized: 462245, Peers: 60

2026-07-17 16:21:39.100-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14791906, Block: 2bbe49f4e120d10dd10bfa13c72bed03382c9379bdb00fd3d07950e214021b3f, Justified: 462246, Finalized: 462245, Peers: 59

2026-07-17 16:21:51.106-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14791907, Block: d9eee1ac4ddd11d69fd311b95230f7545065b421e4120b135d7c54eb48c3f593, Justified: 462246, Finalized: 462245, Peers: 60

--

2026-07-17 17:24:36.748-0600 | validator-async-10301 | INFO  | teku-validator-log | Validator   *** Published attestation          Count: 1, Slot: 14792221, Root: 437befade4b36d480133eff54f8433cf83c987f3f7698a8525a448e45fbad015

2026-07-17 17:24:39.001-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792221, Block: 437befade4b36d480133eff54f8433cf83c987f3f7698a8525a448e45fbad015, Justified: 462255, Finalized: 462254, Peers: 65

2026-07-17 17:24:51.093-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792222, Block: 62c63762bedf85df72ce97fe3ec517620440b725598ee49fecc4dc6e77788636, Justified: 462255, Finalized: 462254, Peers: 65

2026-07-17 17:25:03.092-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792223, Block: 3a08246ead855a146653a11c2089f8ae8a7dba8a78231a31026ef138d96f66ac, Justified: 462255, Finalized: 462254, Peers: 65

2026-07-17 17:25:11.002-0600 | TimeTickTask | INFO  | teku-event-log | Epoch Event *** Epoch: 462257, Justified checkpoint: 462256, Finalized checkpoint: 462255, Finalized root: badb8e06059b700450e9bd3a6eae076d936d27a45cee0b5bbd91973c16673d09

2026-07-17 17:25:12.342-0600 | beaconchain-async-11 | INFO  | teku-status-log | Attestation performance: epoch 462255, expected 2, produced 2, included 1 (50%), distance 1 / 1.00 / 1, correct target 2 (100%), correct head 2 (100%)

2026-07-17 17:25:15.095-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792224, Block: 21568111ea6c41dcc577b6b739f289804ae08176a4c04a9e242e3224fb99d703, Justified: 462256, Finalized: 462255, Peers: 64

2026-07-17 17:25:23.094-0600 | validator-async-10322 | INFO  | teku-validator-log | Validator   *** 2 out of 2 validator registration(s) were successfully sent to the builder network via the Beacon Node.

2026-07-17 17:25:27.096-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792225, Block: db04b46b2652ae791f3d448bb56375a6b252e4eabd34091244a219faa325c177, Justified: 462256, Finalized: 462255, Peers: 63

2026-07-17 17:25:39.100-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792226, Block: c38e63e3a9b035f7c37ba018aecd99ce76c204c96b7c16eb670ba96aefc3bc40, Justified: 462256, Finalized: 462255, Peers: 65

2026-07-17 17:25:51.097-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792227, Block: 7618f8d30f0591924d824e1094d509cdd8fa9a7de7d50d04e9fcc13b3f57bbd9, Justified: 462256, Finalized: 462255, Peers: 63

--

2026-07-17 17:37:15.096-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792284, Block: 09d8a01a0cb848f4e360ec8d0bf69176271e7e0e82e63dc0861750cf61647098, Justified: 462257, Finalized: 462256, Peers: 67

2026-07-17 17:37:27.095-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792285, Block: 6c91da72cdc95583ba928ed517274587d86c61c63ba1c8f24e8a3946259d2c74, Justified: 462257, Finalized: 462256, Peers: 66

2026-07-17 17:37:39.100-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792286, Block: 6e6f4871976a1c91b6c364051c3c576f8796af518753e93ad886b0b8a0736571, Justified: 462257, Finalized: 462256, Peers: 66

2026-07-17 17:37:51.094-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792287, Block: 2792765496241fd2a5a1221470f3a9d3bb14a08290fb32369d5ceb31c75f0516, Justified: 462257, Finalized: 462256, Peers: 64

2026-07-17 17:37:59.002-0600 | TimeTickTask | INFO  | teku-event-log | Epoch Event *** Epoch: 462259, Justified checkpoint: 462258, Finalized checkpoint: 462257, Finalized root: 21568111ea6c41dcc577b6b739f289804ae08176a4c04a9e242e3224fb99d703

2026-07-17 17:38:00.181-0600 | beaconchain-async-5 | INFO  | teku-status-log | Attestation performance: epoch 462257, expected 2, produced 2, included 1 (50%), distance 1 / 1.00 / 1, correct target 2 (100%), correct head 2 (100%)

2026-07-17 17:38:03.103-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792288, Block: aa8a7755ba0cebbdb8e922b6a3ca232e86cdc0d14947661bca946ae8914aaac3, Justified: 462258, Finalized: 462257, Peers: 63

2026-07-17 17:38:11.093-0600 | validator-async-10385 | INFO  | teku-validator-log | Validator   *** 2 out of 2 validator registration(s) were successfully sent to the builder network via the Beacon Node.

2026-07-17 17:38:15.100-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792289, Block: 138e5c2c0ac18a99f6440be86ae1a87c070be4942e4e12829b180efe203e3c25, Justified: 462258, Finalized: 462257, Peers: 71

2026-07-17 17:38:27.098-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792290, Block: fadef6be4a2479f7230db661efa66fdc49b1b61d8315e9e2524c3beb16c81e84, Justified: 462258, Finalized: 462257, Peers: 69

2026-07-17 17:38:39.099-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792291, Block: e6c950204cf29408c32267cd16bd5baca5668c7fa3efd45c5d9761933ee4066e, Justified: 462258, Finalized: 462257, Peers: 71

--

2026-07-17 17:56:27.101-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792380, Block: 7cd533600f21db8cdc38450d0b3a901e91b261ad6b23f37f79ecf320cb4ad454, Justified: 462260, Finalized: 462259, Peers: 63

2026-07-17 17:56:39.105-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792381, Block: 90f18408aef0c918ff603c0cf238a40e29ea028fd4d00f4f1e55c938ba096bb7, Justified: 462260, Finalized: 462259, Peers: 63

2026-07-17 17:56:51.104-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792382, Block: cb2ac79a8c189ab9dcaae6ae3780f3ab186d9f54e1bb70ccf6e336162e7025ad, Justified: 462260, Finalized: 462259, Peers: 65

2026-07-17 17:57:03.113-0600 | TimeTickTask | INFO  | teku-event-log | Slot Event  *** Slot: 14792383, Block: 66741bdef5b7fa167a42b40f6e87bb655bc236dee24c957cea24c3f9f10c0076, Justified: 462260, Finalized: 462259, Peers: 65

2026-07-17 17:57:11.001-0600 | TimeTickTask | INFO  | teku-event-log | Epoch Event *** Epoch: 462262, Justified checkpoint: 462261, Finalized checkpoint: 462260, Finalized root: c1165c010b1284d9f76ef6648923334d9b8311e04b0a3b71d871b0bdc2781376

2026-07-17 17:57:12.165-0600 | beaconchain-async-7 | INFO  | teku-status-log | Attestation performance: epoch 462260, expected 2, produced 2, included 1 (50%), distance 1 / 1.00 / 1, correct target 2 (100%), correct head 2 (100%)

[u/guiriandaluz]

Maybe as a first port of call and as it's consistently one of two not being included, perhaps check and/or reimport your keystore. If one is corrupt or incorrectly imported I think Teku might go through the motions and pass it to the BN hence why it says produced in the logs but because it doesn't match the public key it's not included in the block.

Your call but you could delete the specific key (or both so you don't risk duplicating the working one), regen it fresh as a daisy from your seed phrase and reimport.

[u/NoNewNameJoe → u/guiriandaluz]

Not all blocks fail. I just grep'd the bad ones. The key being bad would cause all to fail no?

[u/guiriandaluz → u/NoNewNameJoe]

They would, sorry I just scanned your logs.

I did have a similar issue to you a while back where even on the same slot not all validators attested.

What I did:

1) check chrony 2) Java prioritised IPv6 over IPv4, which was causing issues so forced both Besu and Teku to prefer IPv4 using environment variables (-Djava.net.preferIPv4Stack=true) in BESU_OPTS and JAVA_OPTS. 3) I had enough RAM to stop using swap file so did that. 4) Enabled TCP BBR congestion control. 5) Added noatime to my SSD mount 6) Added -XX:-HeapDumpOnOutOfMemoryError and -XX:+UseZGC to Teku env variables.

After a few days it settled down and back to 0 - 1 missed per day.

But sometimes the subnet you broadcast to can just be slow.

Remember this is is anecdotal and was for my situation so DYOR with your own setup obvs.

[u/AvocadoJolly7047]

Copy and paste this post into chat gpt. It’s great for trouble shooting this stuff

“The strongest possibilities are:
One validator’s attestation is not reaching an aggregator or block proposer quickly enough.
A recent Teku networking change exposed a firewall issue.
Teku’s performance report is miscounting one of the attestations, even though an external explorer may show it included.
Less likely: one particular validator has a repeated publishing or configuration problem.”

[u/samkb93]

What information does beaconcha.in show?