I will take a very biased perspective and base my answer on only this argument's impact on Tech. I think you should be able to play your games whenever and wherever you want. If your school wants to make more money, play closer to a large fan base or lose a home game off of your schedule, it should be their right. Let's start with more money....For a school like Tech to compete with schools like Texas, A&M and Oklahoma, they need money to help even the playing field. Playing a neutral site game against a conference foe pays out every year, not every other year like a home and home series does. Texas Tech is guaranteed a $2 million payout for each year in the Baylor series taking place in Arlington. Per an LAJ article last year, Tech AD Kirby Hocutt says Tech makes about $3 million per home sellout, so Tech is making $1 million more over the course of a two-year span. That difference adds about 7% to Tech's annual athletic budget. Tech (and any team for that matter) should have the option if they can find an opponent who agrees to play them.
Neutral site games eliminate a home game from the conference schedule of both teams. I enjoy the fact that Tech doesn't have to trek to Waco for a game every other season even if that means the tradeoff of not having a home game every other season. While Texas and Oklahoma never have to play in the unfriendly confines of each other's stadium, they also don't get to take advantage of playing in their own friendly confines, so any advantage is neutralized as is the disadvantage.
Lastly, for Tech and teams in smaller markets, it gives the schools a chance to play closer to their fan base and recruits. Unfortunately not every fan can do a weekend road six or seven times a year so the option to give them a chance to see their favorite school in person is a good thing. The game may also make t-shirt fans out of some people in the market which Tech could use all the t-shirt fans they can get their hands on. It is also nice for teams with smaller athletic budgets to maximize their dollars by hosting recruits in these larger markets when they are noted as the "home" team at neutral site games. This saves some travel expenses, not a ton but every dollar counts for Tech when in an arms race against richer opponents.
Sure as a fan, I don't like giving up the home field advantage every other year to a conference opponent. That advantage could be the difference in a win, a bowl appearance or a conference title, but I feel like the good outweighs the bad in this case. As long as everyone is given the same option of playing in neutral site conference games, then it is fair and every school can decide what is right for them.
The one aspect of college sports that make the events unique and special is the atmosphere at the games and the experience for the fans. For the majority, professional events do not seem to have this intangible trait that college football games excel at. Conference games, especially those against higher-level rival teams, are a prime example of this. Having a big conference game at a neutral site diminishes this effect most of the time.
The Red River Rivalry game is one exception to this rule. The atmosphere in Fair Park is electric every year when Dallas turns burnt orange versus crimson and cream. This experience has become special due to the intense rivalry between the schools and it works at a neutral site. The historic game has become a part of the culture of both teams as they fight for bragging rights for another year. However, with Tech, there is no clear cut rivalry that makes a neutral site game compare with the experience at a stadium on a school's campus. AT&T stadium in Arlington has never had the feel of a big game during our (insert insurance company) shootout game against Baylor. Whether it's the stadium itself, the lack of big meaningful games against the Bears, or the 350 miles that separate the schools, the game lacks that "it factor" that college games have on campus. Attending a game with 60,000 of your closest friends on a clear fall night on the South Plains will always have a better atmosphere than a game inside the monstrosity of Cowboys Stadium.
As important as conference games are, you want your team to have the best chance to win, and having the games at home is the best way for that. Neutral site games at the beginning of the season are all about the media spectacle on ESPN and money for the schools and the stadiums. Yes, the money is a great thing for the schools, and can contribute towards a large chuck of their budget, but for a game that is all about the fan experience, keeping the big conference games on campus maximizes that experience.