There are different aspects to this. The first and the easily verifiable one is that they default to client-server-client connections, not end-to-end encryption. If you want to have an end-to-end encrypted channel, you have to explicitly open a "secret chat". However, this removes the convenience of cross device syncing.
The second one is more difficult to evaluate. If you use the above mentioned "secret chat" feature, Telegram employs their own closed-source encryption scheme. That's usually an indicator to be cautious from the get-go. Since it's closed source, it can't really be trusted.
Encryption for secret chats doesn't involve server, so technically it can be analyzed.
It's a pity Telegram decided to roll their own encryption scheme. I use Telegram a lot for daily business because it's superior desktop messenger product. I would gladly participate if somebody started a crowd-funding for Telegram's security and encryption audit.
Can someone elaborate?