The people cannot come up. Moses can only have meant, that the people could not approach the mount unwittingly, since the fence commanded (Exodus 19:12) was made. But to scale the fence, or break through it, was of course possible.
(See Exodus 19:13.