Bank 1 has entered into a CP with you regarding a specific debt.
As long as you maintain your end of bargain, they can not and will not exercise their Right to Setoff.
Under common law that ability clearly exists and is an option should you not make your payments as arranged.
The reason I mention different branches is many confuse that for different bank which it is not.
You have a legal CP in place and they can not alter those terms at this point.....unless you breach that agreement by not paying on time.
So simple answer is no they can not take your deposit unless you screw up your CP terms with them.