In Oregon, the votes are received by teams of people that are required to have representatives from each of the major parties. They check signatures, then open the outer envelope and take out the secrecy envelope which has the ballot inside and toss that in another box. The person checking the signature does not do anything else with the ballot itself. Then the pile of ballots in their secrecy envelopes is given to another team and they open the envelope and process the ballot.
Yes, the mailman could commit a federal crime and figure out how you voted for. So if you're worried about that, drop the envelope in the actual ballot box.
Your address, or your name, will be enough to determine how you most likely voted.
The mailman sees even more. If you live in a condo and drive a Prius with a "Coexist" sticker, you probably voted one way. If you live in a single-family home with a big yard and you drive a pickup with a "Blue lives matter" license plate holder, you probably voted the other way.
Yes, the mailman could commit a federal crime and figure out how you voted for. So if you're worried about that, drop the envelope in the actual ballot box.