The conventional wisdom in an election is that any candidate from a minor right wing party (Libertarian, Constitution) hurts Republicans and those from a minor left wing party (Green, Working Families) hurt Democrats. The voter is finding an alternative ideologically and would vote for the major party candidate if the minor party candidate weren't on the ballot.
This line of thinking is due to people assuming that a third party vote is ideological. More often than not it's a voter who just doesn't like the major party candidates and chooses anyone else regardless of party. They are just looking for a name. If a Libertarian is the only alternative to the Republican and Democrat that candidate could take more voters from either of the major party candidates. People really dislike both Hillary Clinton and Donald Trump and Libertarian nominee Gary Johnson could pull from either. This Democracy Corps survey suggests Johnson is pulling more from Clinton. (Look on page three)
Who are these people? They could be primarily a mix of two groups. The first are Bernie voters who'd only vote for Clinton if she were the only alternative to Trump. They'd vote for someone they disagreed in order to not vote for Clinton. The other group could be Republican #NeverTrump people. They dislike Trump so much that they'll support Clinton. Yet given a third party alternative, and a right wing one at that, they'll go to Johnson. The left wing alternative to Gary Johnson is Green party candidate Jill Stein. She has the problem of being lesser known than Johnson, a two term New Mexico governor, but also not being on the ballot in many states.
On election day the third party candidates might or might not take a significant share of the vote. If they do, we shouldn't be so quick to assume who they'll be taking it from.