Ideally, it should all be broken up so that no lumps are bigger than 50-60mm. If you have lots of big lumps, you can end up with significant differential settlement within the sub-base, which is then reflected in the paving.
To put that in layman's terms, the smaller bits in the sub-base may settle by, say, 10%, but the big lumps will not settle, or will settle only, say, 3%, so that what was a level sub-base soon becomes an uneven sub-base, and this difference in settlement is carried through to your paving, turning a level pavement into an uneven surface.
If you're intent on using the old concrete as a fill/sub-base, it's really worth the extra effort of breaking it up properly, with a sledgehammer, if necessary and getting a good, even sub-base. The analogy I always use is that of an underlay beneath a carpet - it doesn't matter how much you spend on the carpet and how well it is laid, if the underlay is crap, the final carpetting will be crap too.