Accurate crop type and crop growth stage maps are essential for agricultural monitoring and ensuring food security. A wide variety of airborne and spaceborne sensors now provide high spatial, spectral, and temporal resolution images, which are vital for crop mapping and monitoring. Crop type and its growth stage can be characterized by spectral, spatial, and temporal features. The classification of crop types and growth stages has been explored in previous studies as independent tasks. However, the growth stages of a crop are an important factor in identifying the crop and vice-versa. A multi-task learning (MTL) framework is proposed in this work to classify the crop type and its growth stages simultaneously. A hybrid convolutional neural network and temporal convolutional network (CNN-TCN) architecture is presented to process a multitude of features relevant to the tasks. To learn the spatio–spectral features, we fed the hyperspectral input to 3D convolution blocks and multispectral input was given into 2D convolution blocks. We reformulate these multi-channel features into two dimensions and feed them into the temporal convolutional neural network. Subsequently, we use two fully connected branches for each task. MTL frameworks were developed for multispectral (Mx), hyperspectral (Hx), and the combination of Hx and Mx (Hx-Mx) images to model crop type and crop growth stage classification. Results reveal that the proposed model for Hx-Mx outperformed the best single-task model by 13% and 8% in crop growth stage and crop type classification, respectively. Compared to single-task models, the proposed model can exploit the high spectral information from Hx images and high spatial information from Mx images, making the proposed model more useful for unmanned-aerial-vehicle-based crop mapping.