Augur's reporting fee system is designed to put upward price pressure on REP such that the REP market cap remains at least 5x larger than Augur outstanding interest. What is the reasoning between 5x over any other multiplier?
In order for REP to be secure against a 51% attack, the system tries to ensure that an attacker cannot make more money from a successful attack on the system than it would cost them to execute the attack. In order to win a fork, (assuming everyone is perfectly efficient) the attacker must buy more than 50% (often referred to as 51% for simplicity) of REP to force the fork to resolve incorrectly. Afterwards, the market will likely assign no value to the attacker's fork of REP and assign notable value to the remaining un-forked REP. This means that the attacker will effectively lose all of the money they used to buy up the 50% of REP required to launch the attack. This gives us the cost of the attack as being 50% of REP market cap.
Again assuming perfectly efficient actors (worst case scenario), the attacker's best strategy would be to buy incorrect shares in every market for epsilon (very small number) dollars each, then resolve all of those markets incorrectly and redeem the winning (but incorrect) shares for full share value. This means the most the attacker can make from this attack is all of the open interest across all current Augur markets (but not future markets).
Now that we have established the maximum profit from an attack (Augur open interest) and the minimum cost of an attack (50% of REP market cap), we can assert that in order for the system to ensure that an attack of this style is not profitable (still assuming everything is perfectly efficient) we need to ensure that REP market cap is 2 times larger than Augur open interest.
Now, in reality things aren't perfectly efficient, especially in blockchains with long block times and many programming constraints. Both the attacker is inefficient and the defenders are inefficient. Because of this, we want to give some wiggle room for the security model so that it can still secure the system even in the face of these inefficiencies. To achieve this shock/inefficiency resistance we need to increase the multiplier to something greater than 2x.
Why not 100x? The higher the number, the more traders have to pay for security. If the traders are overpaying for security by too much, they will leave the platform for a competing platform that more appropriately charges for security. This gives us some downward pressure on this number.
The development team felt that 5x was sufficiently bigger than 2x such that there should be no problem withstanding shocks to the system or defense inefficiencies while also not being so large as to be an undue burden on traders.