After checking a lot of cases we did the next:
we compared smtp connection via 25 port on new server with old
openssl s_client -connect old.post:25 -starttls smtp
openssl s_client -connect new.post:25 -starttls smtp
and we obtained, that old server answers that he don't support any starttls, but new server answers, that we use incorrect certificate, and that our certif mismatch with him and wrote in the error the server certifate that we need. we copy this information from this error, and generated assembled by hand new certificate. After adding him to all repositories and restarting application, message sending started to work.
So, we think that the core mail java lib, somehow checks this information about 25 port, and therefore generated a request with certificate searching by option mail.smtp.starttls.enable=true. because this port also is open, and supports any connections without using any certificates.