We present a robust vertebral segmentation framework that can bootstrap the segmentation task with little to no dataset. Our deformable model-based framework jointly optimizes the appearance and shape of the spine model using the novel differentiable appearance modeling method. Because our framework learns the appearance of the spine only from the given image, it does not rely on the dataset or handcrafted image features and adapts robustly to the appearance of the image. With our proposed differentiable signed distance operator and spectral mesh optimization, the shape of the spine model can be refined via a gradient-based optimizer. Our framework was tested on the VerSe'20 training dataset, and it achieved an average Dice score of up to 90% for selected vertebral labels. Our results suggest that utilizing the explicit knowledge from the template model can significantly reduce the need for a large training dataset.