Wow, this is pretty crazy. Just by using opacity: 0; pointer-events: none instead of display: none, I was able to get dragging on mobile from almost unusably laggy to a barely noticeable amount of lag.