Having played tf2 and rollback fighting games at 150 ping I'd say that they are similarly playable. Projectile is tougher than fighting games but hitscan is probably a bit easier so about the same overall. If I'm not mistaken tf2(and most source games/most fps games in general) do use a form of rollback. The game predicts where enemies will be and when it's wrong it corrects it. Difference is in fighting games when it corrects the other character essentially teleports to their new position/animation and source smoothly moves the character there.

There are a ton of people that know a lot more about how these things work than me so please feel free to correct me/expand.

Thanks for posting.

So for some reason it's not allowed to predict spawning a projectile. I don't get why, especially since the lag compensation system on the server is already able to go back in time to figure out when a command happened

Furthermore, I cannot find on that page what is actually inside a snapshot. Does it have player positions or inputs(besides things like entity spawns to draw rockets and play weapon sounds?) ?
How does the client predicts where the other players will be on the next tick?

Also the GGPO document mentions that games have to be 'deterministic' in order to use it. Is source engine deterministic?

Rollback only works in a Peer to Peer system because the whole nature of it is that it allows users to input their inputs without having to worry about delay, making execution much more consistent. The main benefit of the system for fighting games is that it lets your execution be uninterrupted, as dropping a combo holds much more substantial weight than missing a shot 99% of the time. But since general execution is much more often prioritized in fighting games and the fact that a majority of the time you aren't actually inputting any specific inputs, rollback works perfectly for fighting games. In Source games, you aren't required to hit sometimes frame-perfect inputs constantly to execute what you want to like in fighting games.

In fighting game rollback, they don't necessarily "predict" anything, they simply continue to hold down the previous input. This works out in a lot of fighting games due to the amount of time where you're not actually able to act (blockstun/hitstun when hitting or blocking moves, jump start up frames, etc.). Without a machine learning system it's pretty much impossible to genuinely predict any future inputs. In an FPS game its much more detrimental to have that happen since you never really have "dead time" where you can't input anything.

As for predicting projectiles, it can't simply predict spawning a projectile because it would mean predicting a spawn on every missing packet, and then retroactively correcting that input, despawning the rocket and undoing any damage that might have happened, on every client connected to the server. The amount of load that would bring on the server, let alone the individual clients constantly correcting itself would become way too much to handle for 12 different players. Damage calculation is just rerunning some cached numbers, whereas projectiles require physics calculations and actual entity spawning, etc.

With random spread disabled, probably.
The point is anything that involves a random number generator would be a problem since everyone would get different results.

I'd guess they don't want to deal with the mess that that creates. Firstly allowing the client to predict a projectile that may or may not ever be created in case the player dies before it actually happens but doesn't know it yet (the joys of latency) means they have to clean up "dead" projectiles. Projectiles seeming to spawn and then being deleted is also going to create a lot more salt than projectiles always spawning late if your ping is shit.
Secondly if the time of creation is based on what the shooter sees then everyone else is only going to see them after they have already traveled for ping from shooter to server + processing time + ping from server to everyone else. In other words: teleporting projectiles. Would you rather have someone with 900 ping have a suboptimal experience or have the projectiles he shoots suddenly appear in someone's ass without any chance of dodging?

They don't sent state data, they send only inputs, which cuts down a bit on the bandwidth but most importantly peer-to-peer means instead of the client taking the inputs, rendering a frame (with predictions and data from the server for what everyone else does), sending their inputs to the server, the server rendering a frame, then sending the current game state to everyone who then renders the frame again they can instead send the inputs straight to everyone who can then start rendering their frames. That removes about half the latency.
They also don't have any centralized game state so every time a prediction turns out to be wrong they have to instead execute all frames from the point of divergence again with the right inputs without rendering it visually to catch up to the correct state.
Last but not least there is no agreement as such, it is simply expected that everyone will arrive at the same state with the same inputs. That requires that no one deviates accidentally (no RNG allowed) or maliciously (cheating). No idea how they deal with packet loss. Allowing resynchronization by simply sending your own state/position sounds very exploitable.

Isn't it what "client side prediction" is?

What I meant by predicting projectiles would be to do exactly what is done with client side prediction, i.e. let the player do his inputs first and (maybe) roll them back later.
Because this could let a player perform a rocket jump and feel the blast effect immediately instead of having it delayed by whatever lag there is.

I am not talking about predicting that an opponent is going to spawn a projectile at your face.

I did not understand that part. If we agree that a client cant "predict spawn" projectiles of opponents, how is that an issue? A client would still see on its display whatever entities the server tells him exist.

Yes, this would be a problem.
In this GGPO article https://drive.google.com/file/d/1cV0fY8e_SC1hIFF5E1rT8XRVRzPjU8W9/view the dev explains that in practice it can hide the move starting late during the startup frames. We don't have that in shooters.

However, just as GGPO adds a fixed number of frame delay to mitigate for this, I believe what is rendered on client side in source is "the past", for the server :

The way I understand that, all clients are more or less in the past depending on their interp value. So if everyone agrees to be a fixed amount of time in the past, and that amount is great enough to mitigate teleporting projectiles issues, this problem should not arise most of the time.
I believe this is why interp is 100ms by default as well, if only jumping wasnt so sluggish, and everyone was forced to the same value, there would be no downsides to keeping it

Quoting this article :
http://mauve.mizuumi.net/2012/07/05/understanding-fighting-game-networking.html

-> looks like a client must keep some history of past input data sent, to be able to give it to another client that lost a packet.

I think you're fundamentally missing where the prediction happens. The prediction that happens in peer-to-peer rollback is for missing inputs of OTHER players based on the last received input. With two clients, Client A and B, if Client B drops packets it was supposed to send to Client A, rollback netcode takes the last known inputs of Client B that it receives, and Client A then predicts missing data based on that. For rollback to work on a source server setup, the server would have to do that same "prediction" (which would most likely just be a movement direction), and then send that prediction to every client except for the one dropping packets. So the other 11 players would receive predictions from the server and display that, and then once the new inputs are received, they would be retroactively synchronized into the game state.

Shifting game state calculations away from the server and onto each individual client is also a theoretical way of implementing that, but when managing a 1 on 1 peer to peer synchronization is already hard enough, you would essentially leave the server with the task of managing a 12 or 18 person way essentially peer to peer connection where if one person's game state is slightly different from the rest the server has to send them corrected data from every other client, which just generates WAY more input delay due to the increased amount of traffic that would be involved.

The 50ms mentioned in the documentation is separate from the ping you have to the server as well afaik, as the server has to receive an initial input (lets say at an example 40ms ping), allow the 50ms buffer to receive the next snapshot, and then start allowing clients to render afterwards. Fighting games don't have this extra rendering delay due to being locked at 60fps, and they simply send an input for each frame.

For source/general fps games, the equivalent of fighting games' 2-3 frame delay is like everyone playing at a set buffer based on say 50 ping. If you set everyone to 50 ping buffer on a server, it doesn't magically make the person at 150 ping send his inputs any faster, and that person will still have to send their inputs and data 100ms later. In a fighting game, you can cover up that extra delay by interpolating the new animations through the startup frames that are involved in pretty much every action you do. However in any FPS game, you very rarely have any example of startup, outside of something like switching weapons where you are inactionable. When you click, your gun shoots immediately, there's no room to interpolate the old action with the new. Rollback means you would have projectiles be spawning halfway through their travel time on people's screens, or have people suddenly teleport halfway into their rocket jump, which already does happen.

The only surefire way to avoid having something like this happen, would be to have the input delay be equal to that of the highest ping player on the server ALONGSIDE a rendering buffer of the same length, and if everyone else in the server has 40-60 while one person has 150+ ping, it makes absolutely 0 sense to punish everybody else.

If the client can only see the entities the server already knows about it can't "predict spawn" its own projectiles either.

They don't add a delay. They do teleporting/warping by simply jumping to the actual position and because the only thing that will be skipped with decent ping are the startup frames they call it good enough.
Note that this is a horrible idea in any game with hitscan weapons. It's not ideal in their case either because you're losing a couple frames worth of reaction time, which was kind of the point of having startup frames instead of the damage starting immediately but there's always drawbacks one way or the other.
The whole selling point of their method is that there is no delay to your actions, making it feel just like a local game and the tradeoff is that if prediction fails (that is part two of their strategy) you get warped to the correct state but as long as prediction is right everything is exactly as it would be in a local game.

Either projectiles spawn instantly locally and teleport for everyone else (GGPO, if prediction fails) or they appear everywhere at the same time at their actual spawn point which means they guy who pressed the button has to wait until everyone knows about the projectiles before they get to see their own projectile (source). You can't have both at the same time.

You can treat rng in the same way as player inputs (which you can't predict), and the whole problem disappears.. even better if you treat the rng seeding as player input. Though it might be technically easier to synchronize the rng by having the server pre-determine the random results before they're needed, (ex. "the next time this scout fires the scattergun the bullets will come out in this pattern") and sending every client the info.

A much bigger problem may be that small hitches in the game's performance seem to have small effects on things like player trajectories. normal rng works deterministically and can be 'predicted', but predicting exactly how the engine interacts with every player's pc might be a nightmare

For me, source prediction happens on the client, based on what was last given by the server.

Nah, the server doesnt care about this stuff. it computes a game state however it likes, then sends that game state to every client. What counts is that what is displayed on each client is as smooth and reactive as possible.
Therefore here I am not discussing "shifting away game state calculations to clients", I agree that it's not feasible and pointless.

Thats the point of having everyone use the same interp value : everyone would see the prjojectile appear at the same place. The lower the interp, the farther from the shooter the projectile will appear.
Hence, with everyone at high interp and playing in the past, the downside of the projectile spawning farther away could be mitigated, as everyone would be rendering frames related to the same tick and have an array of more recent ticks received stored waiting for eventual rollbacks. So if that array is big enough compared to the connection quality of the worst player, all the netcode shenanigans are resolved before anyone renders the tick, and every client renders the same thing.

With all that you said I still dont see how this wouldnt work.

I am not talking about any delay-based netcode such as what is done in fighting games not using rollback, because that's out for FPS for reasons stated in the valve documentation.

Why not? I believe most of the time when a client requests to spawn an projectile it is allowed to do so. If the client dies after predict spawning a projectile but before the projectile existence is confirmed by the server, then only that dead client will see some kind of visual artifact. And that client's dead anyway, no big deal.

I thought I've heard the KI dev talk about it but here is smth else instead :
https://arstechnica.com/gaming/2019/10/explaining-how-fighting-games-use-delay-based-and-rollback-netcode/

Could you elaborate on why this is a bad idea for hitscan things?

Sure, my question now is, if it's the first case, can you mitigate how much the projectiles teleport by increasing what i believe is named interp, which is delay between what state is computed by server and what state is rendered by clients?

Rollback netcode at its root is a delay based netcode system though, only due to the ability to rollback dropped packets they can set a low fixed delay instead of requiring a variable delay in order to ensure the inputs go through properly. By using inactionable frames, rollback can seamlessly interpolate the old game state with the actual game state.

The interp can only account for so much when the person is still 100ms behind everyone else, unless you also want to force everyone else to play at a noticeably higher input delay. The larger the difference between the interp and the ping, the more rollbacks will occur. Rollback's largest benefit is that it helps deal with packet loss, not general large ping differences like you first mentioned. Sure your inputs feel like offline, but that doesn't mean the person with 150 ping still isn't laggy as fuck and teleporting fairly often. In Melee, which probably has the most similar pace to TF2, 150 ping 3 frame is barely playable against any mobile character as they start micro teleporting every other movement and you have to try and pin them down.

Since all hit detection occurs server-side, if a client predicts the movement of someone else incorrectly, than they'll simply be shooting at ghosts. Alternatively if you were to change lag compensation to reflect compensating for false predictions, you would have people getting hit for moving in a way they never did. In the first scenario, a scout will be shooting at the ghost of a soldier who's already rocket jumped, and then have to react to them teleporting and adjust to hit them as they fly towards them much closer. In the latter scenario, a scout will win a 1v1 solely because the other scout dropped a packet and started running in a straight line.

Okay. I can't say I'm out of the fog but I learned things thanks to you, so cheers =)

No problem man, if you ever wanna discuss it more I'm down, netcode is a really interesting topic to think about for me. Glad you learned some things from all the talk

There's a fundamental misunderstanding here. Source does not predict, it interpolates because it already knows what's going to happen. The client is always so far behind the latest packet it recieved (well, not if the interp is too low) that it does not have to predict the future. E.g. the client got packets from time t and t+1 and is rendering time t+0.5. It doesn't have to guess where everything will be, it can interpolate that it'll be halfway between the positions at 0 and 1. By the time the client gets around to rendering t+1 it should have recieved the packet from t+2. This would be interp_ratio 1, the bare minimum, usually it's even more packets so the guess can be better than "halfway" and account for changing movement speeds.

Yes, it's much less of a problem in games without stuns but you still have to clean up the dead projectiles (more code), generate more salt because the player sees his own projectiles spawning and then disappearing and the teleporting problem still exists. It's not worth the trouble.

No.
If everyone is living in the past and a projectile instantly spawns locally then it has spawned in the past exactly as far behind the present as the player is. Except no one but the player knows about it. Everyone else will see it after the usual delay that ping/transmission time and server processing time adds. If you want everyone to see the same thing then at that time it must appear where it has already traveled to on the shooters side, meaning it has to teleport.

frootloops already answered the technical side but I think I should add something you're missing: These are completely different use cases.
Fighting games have predetermined animations that will always play out the same and fixed framerates.
TF2, or any fps for that matter, has neither. Any movement can be canceled or even reversed at will. That is a fundamental requirement for dodging. In a fighting game you get an input an know what will happen for the next 20 frames. Fps don't get that. Anything could happen in the next frame. Any attempt to predict when which input will happen is doomed to fail.
On top of that no one will have the same framerate. A delay of 3 frames is meaningless when someone could be dropping to 30 fps and someone else might still be at 300 fps. You're not even in sync with the server's fps. So instead you get another problem: You can't use the server's snapshots directly. Your 144 fps are not in sync with the server's 66.6667. You will not get a snapshot for every frame you render and even when you do get one it isn't necessarily from the exact same time that your current frame will be. So you need to guess for possible states between the servers snapshots. Prediction is useless so extrapolating from past snapshots towards the future is a fool's errand and should only be done when there's no other choice because guessing movement will continue as is is at least better than freezing everything. So what do you do? You don't render the latest snapshot, you put your personal time at least one snapshot behind the latest one recieved and interpolate between it and older snapshots.

The end result is that you have to interpolate and you can't predict others' inputs so you might as well use interpolation for that as well. (see first paragraph)

Also something I should've added when you asked about TF2 being deterministic, but thought it would be complicated and unnecessary in this case. Thankfully Taat (#9) reminded me that it is relevant in this case.
TF2 should be deterministic, but isn't in practice because the source engine's physic calculations are fps dependent. Now that isn't good, but it is unavoidable because when your fps fluctuate and aren't in sync with anything sometimes you have to calculate 0.0034782378623425 of the movement per second because that's just what fraction your framerate is and there's only some much precision you have and then the rounding errors add up. This isn't a problem in a server based game because the errors are tiny and the server can assert its authority and bring everyone back on the "right" track (the slightly wrong track it has chosen instead of the other slightly differently wrong tracks everyone else has chosen) by the next snapshot. In a peer-to-peer game that assumes every input has to same result from everyone's perspective however you are fucked. Everyone's game state would drift further and further apart. Even if you try to synchronize regularly there's no way to decide on a "correct" state. Everyone is wrong, just in different ways.

tl;dr
Fighting games and fps are fundamentally different use cases, using netcode made for one in the other would have terrible results.