Objective To prepare human SET8 monoclonal antibody and explore its effects on the proliferation, apoptosis, and cell cycle of hepatoma cells, and to evaluate its anti-tumor effect in mouse models of hepatocellular carcinoma. Methods We immunized mice with human SET8 polypeptide fragment and screened and fused B cells and myeloma cells to establish a hybridoma cell line that stably secreted SET8 monoclonal antibody. Production was expanded by intraperitoneal injection into mice and the collection and purification of ascites. We investigated the effects of SET8 monoclonal antibody on the proliferation, apoptosis, cell cycle, and apoptosis-related protein expression of hepatocellular carcinoma cells by CCK-8, flow cytometry, and Western blot, respectively. Finally, we constructed a mouse model of human hepatocellular carcinoma by cell transplantation to evaluate the inhibitory effect of SET8 monoclonal antibody on tumor growth in vivo. Results Human SET8 monoclonal antibody significantly inhibited the viability of Huh-7 and Mahlavu hepatoma cells at concentrations of 50 and 100 μg/mL, in a concentration-dependent manner (P<0.05). Flow cytometry analysis showed that SET8 monoclonal antibody, paclitaxel, and their combination significantly increased the apoptosis rate of Mahlavu cells compared with the blank control group, with the combination group having the greatest effect (P<0.05). SET8 monoclonal antibody also induced Mahlavu cell cycle arrest in S and G₂ phases and reduced G₁ phase cells. Western blot analysis showed that the monoclonal antibody increased the expression of the apoptosis-related proteins Bax and Caspase-3 (P<0.05). SET8 monoclonal antibody, alone or in combination with paclitaxel, also effectively inhibited the proliferation of hepatocellular carcinoma cells in nude mice, with the combination therapy having the most significant effect (P<0.05). Conclusions The prepared human SET8 monoclonal antibody effectively inhibited the proliferation and promoted the apoptosis of hepatocellular carcinoma cells, and showed good anti-tumor effects in mice.