Travel between both nations dropped sharply overall compared to this time last year, but more Americans visited Canada in June than the other way around.