Referring by this Github Repo, I added CosineAnnealingScheduler with warmup each cycle : PR
For consistency, I add argument in LinearCyclicalScheduler class while preserving BC(backward compatibility) : PR
Also I added test code for above new arguments