Cellulose is an abundant component of the plant biomass in agricultural waste valorization that may be exploited to mitigate the excessive use of synthetic non-biodegradable materials. This work aimed to investigate the cellulose utilized by alkaline extraction with a prior bleaching process from rice straw, corncob, Phulae pineapple leaves, and Phulae pineapple peels. The bleaching and alkaline extraction process was performed using 1.4% acidified sodium chlorite (NaClO2) and 5% potassium hydroxide (KOH) in all the samples. All the samples, without and with the alkaline process, were characterized for their physico-chemical, microstructure, thermal properties and compared to commercial cellulose (COM-C). The extraction yield was the highest in alkaline-extracted cellulose from the corncob (AE-CCC) sample (p < 0.05), compared to the other alkaline-treated samples. The undesired components, including mineral, lignin, and hemicellulose, were lowest in the AE-CCC sample (p < 0.05), compared to raw and alkaline-treated samples. The microstructure displayed the flaky AE-CCC structure that showed a similar visibility in terms of morphology with that of the alkaline-treated pineapple peel cellulose (AE-PPC) and COM-C samples compared to other alkaline-treated samples with a fibrous structure. Fourier Transform Infrared (FTIR) and X-ray Diffraction (XRD) of AE-CCC samples showed the lowest amorphous regions, possibly due to the elimination of hemicellulose and lignin during bleaching and alkaline treatment. The highest crystallinity index obtained in the AE-CCC sample showed a close resemblance with the COM-C sample. Additionally, the AE-CCC sample showed the highest thermal stability, as evidenced by its higher Tonset (334.64 °C), and Tmax (364.67 °C) compared to the COM-C and alkaline-treated samples. Therefore, agricultural wastes after harvesting in the Chiang Rai province of Thailand may be subjected to an alkaline process with a prior bleaching process to yield a higher cellulose content that is free of impurities. Thus, the extracted cellulose could be used as an efficient, eco-friendly, and biodegradable material for packaging applications.