Assume at some point in the future Augur marketcap greatly exceeds that of Ethereum.
In that situation would the current Ethereum PoW or future PoS security model retain sufficient security incentives to protect REP token holders?
There are security measures built into the protocol with respect to keeping the platform secure along these lines (i.e., calculating the open interest on the platform at any given time relative to the price of REP tokens). If the price of REP tokens increases greatly, it would be that much more expensive for someone to successfully attack the platform.
It wouldn't, that said, it's fairly easily fixable. REP and other tokens could be used to stake in PoS as well. What matters is not the market cap of Ether, but the market cap of all tokens used to stake on Ethereum. Today that's just going to be Ether, but in the event of the issue raised in the OP, it'd be fairly straightforward to enable additional staking in ERC20 tokens.