That camera adapter is a multiplexer.
What it means, is that you use software to select which camera you want to get video from and then record it.
It is unclear if the camera has to initialise after you selected it.. or if it is constantly streaming video and all you are doing is "switching channels"
The best application for this is 3D photography. So basically static objects that do not move.
What you are trying to do with 360video will not work. After stitching you will get black holes all over the place and probably induce vomiting.
You may get better results using USB WebCams that encode h264 on the webcam and store the stream on the Pi, download the data to a Xenon 3.8Ghz and stitch it...
Ahhh.. were you thinking you could stream and stitch on the Pi? "No way, José"