The precise mapping of crop spatial distribution using remote sensing datasets is a fundamental task in precision agriculture, which has experienced profound development triggered by the continuous improvement of earth observation systems jointly with the innovation of machine learning theories. However, the extraction and classification of cultivated areas are typically accomplished simultaneously with single-task models at pixel scale, and the prior geographical knowledge was generally ignored, which may present accuracy limitation and significant separation from the monitor application. In response, the proposed work is a novel attempt to address successively the parcel extraction and parcel-wise crop classification over mountainous regions with heterogeneous and fragmented smallholder agriculture. Land parcels get precisely delineated utilizing an improved Densely Connected Link Network (D-LinkNet) with geo-knowledge as prior constraints. Each parcel is then correlated with its closest neighbors considering the environmental and temporal similarity, and classified subsequently by a proposed attention-based Network. Results show that ideal precision was attained in both stages. The incorporation of prior knowledge and neighborhood information has effectively enhanced the accuracy of parcel extraction and crop classification, respectively. Overall, the parcel-wise crop mapping framework may constrain the analysis range within the agricultural space and provides identification results corresponding to real geographic objects, contributing directly to the downstream applications such as crop monitoring, management decision-making, etc.