The meaning behind this simple old saying is that it is the baby which is by far the most important, precious thing. It may be sitting in a pool of filth, scum, and slime. So what? Toss the water, keep the baby.
The same theory can be applied to many things in life, but most especially must always be remembered when dealing with the most important things. One of those most important things is your spiritual life. Christians throughout history have far too often thrown the baby, in this case the Church, out with whatever bath water was mucking things up at any particular time.
In the past, Christians have left the Church, the only one founded by Jesus Christ himself, because they didn't like things that were going on within the hierarchy, or because they didn't believe in some matter of doctrine, or because they had been let down or felt betrayed by some scandal. This is exactly how Protestantism and Orthodoxy began.
The splits, or schisms, within the Church have left many wounds unhealed after centuries, and have left hundreds of millions of true believers in Jesus Christ susceptible to heretical teachings and practices.