Personalized Peptide Vaccine in Treating Patients With Advanced Pancreatic Cancer or Colorectal Cancer

Eligibility Criteria

• * COHORTS A AND B: Patients must have metastatic pancreatic ductal adenocarcinoma (PDA) or metastatic colorectal cancer (CRC) to be eligible (PDA patients with an elevated tumor marker following a primary pancreatic surgery would be eligible)
• * Patients can have any lines (including zero) of prior therapy to sign consent prior to tissue harvest; vaccination will not take place until at least one line of standard chemotherapy is given
• * Patients must have adequate fresh or frozen tissue available or planned to be obtained; for cohort C and D patients should have estimated adequate tumor tissue that is planned to be resected (approximately \> 1 cm cross-sectional size on radiographic imaging); subjects may have tissue collected under protocol PA15-0176
• * Age \>= 18 years
• * Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%)
• * Life expectancy of greater than 6 months (12 months for cohort C and 9 months for cohort D)
• * Absolute neutrophil count (ANC) \>= 1,000/mcL
• * Platelets \>= 75,000/mcL
• * Total bilirubin =\< 2.0 x institutional upper limit of normal
• * Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x institutional upper limit of normal (=\< 5 X if known liver metastases or due to ongoing chemotherapy treatment defined as chemotherapy within 3 weeks prior to lab draw) (except in Gilbert's disease where direct bilirubin will be used)
• * Calculated creatinine clearance \>= 40 mL/min/1.73 m\^2
• * Patients must demonstrate an ability to understand and the willingness to sign a written informed consent document
• * The effects of a peptide-based vaccine, pembrolizumab or APX005M on the developing human fetus are unknown; for this reason, women of child-bearing potential and men must agree to use adequate contraception at study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately; birth control specifications: unless surgically sterile by bilateral tubal ligation or vasectomy of partner(s), sexually active participants must use birth control during and for \> 120 days after the study; abstinence is also an acceptable form of birth control
• * FOR COHORT C ONLY: Patients must have metastatic colorectal cancer (CRC) and are planned to or have undergone complete (R0 or R1) metastectomy/ies (liver or peritoneal or lung or other organ site); the presence of nonspecific lung lesions \< 1 cm are allowed
• * FOR COHORT C ONLY: Agreement to have post-operative blood test to determine plasma mutation ctDNA positivity drawn within 6 weeks following surgical resection
• * FOR COHORT D ONLY: Patients must have localized or metastatic PDA and are planned for complete resection (R0 or R1)
• * FOR COHORT D ONLY: Agreement to have post-operative blood test to determine plasma mutation ctDNA positivity drawn within 6 weeks following surgical resection
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): ECOG performance status 0-1 (Karnofsky \>= 60%)
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): Life expectancy of greater than 6 months
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): Patients must have either measurable disease per Response Criteria in Solid Tumors (RECIST) version (v)1.1 or evaluable disease defined as an elevated tumor biomarker (CA19-9, carcinoembryonic antigen \[CEA\] or ctDNA mutation); pancreatic cancer patients with an elevated tumor marker following a primary pancreatic surgery would be eligible (cohorts A and B only)
• * FOR COHORT C AND D ONLY: Patients must have plasma mutation ctDNA positivity within 6 weeks following surgical resection or ctDNA positivity on any serial testing timepoint if the initial ctDNA timepoint was negative or testing failure occurred.
• * FOR COHORT C AND D ONLY: Completion of all planned adjuvant anti-cancer therapy
• * FOR COHORT C AND D ONLY: Radiographic disease status is not relevant to inclusion for treatment
• * Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome (AIDS); HIV-positive patients on combination antiretroviral therapy are ineligible because of the potential for lack of efficacy of therapeutic cancer vaccine
• * Patients with known brain metastases should be excluded from this clinical trial because of their poor prognosis and because they often develop progressive neurologic dysfunction that would confound the evaluation of neurologic and other adverse events
• * Subjects with active, known or suspected autoimmune disease; subjects with vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll
• * Subjects with a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications; inhaled or topical steroids and adrenal replacement doses \> 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease
• * Uncontrolled concurrent illness including, but not limited to ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• * Women of child bearing potential who are pregnant or breastfeeding; women with a positive pregnancy test at enrollment or prior to administration of vaccine
• * Has history of (non-infectious) pneumonitis that required steroids, evidence of interstitial lung disease or active, non-infectious pneumonitis
• * Known history of active TB (Bacillus tuberculosis)
• * Hypersensitivity to vaccine, pembrolizumab, imiquimod or APX005M or any of its excipients
• * Has a known additional malignancy that is progressing or requires active treatment
• * Active coagulopathy
• * History of arterial thrombosis within 3 months of starting study treatment
• * History of New York Heart Association class 3-4 heart failure or myocardial infarction within 6 months of starting therapy
• * Has a known history of hepatitis B (defined as being known hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (defined as being known hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] positive) infection
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): Patients who have had chemotherapy or radiotherapy within 2 weeks prior to first treatment or those who have not recovered to baseline from adverse events due to agents administered more than 2 weeks earlier (washout period)
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS):
• * Women of child bearing potential who are pregnant or breastfeeding
• * Women with a positive pregnancy test prior to administration of vaccine
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): Uncontrolled concurrent illness including, but not limited to ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): Patients may not be receiving any other investigational agents within 2 weeks prior to first treatment
• * JUST PRIOR TO FIRST VACCINATION (WITHIN 21 DAYS): Has received a live vaccine within 30 days of planned start of study therapy \* Note: Seasonal influenza vaccines and COVID-19 vaccines for injection are generally inactivated flu vaccines and are allowed; however intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines, and are not allowed