Rabbits have a natural instinct to drop their urine and pellets in just one or at least only a few places, often in corners or near their food. Finding an occasional "pill" of poop in their food bowl is their way of marking it as their own. Once a litterbox is being used inside the cage it will be a great tool for the rabbit’s exercise area or even when it lives loose in your home. Some rabbits litter train almost immediately and some take longer, but just be patient and keep trying and you will succeed with most rabbits.
Keep the rabbit and the boxes in one room at first to make things easier for you and the rabbit. A small uncarpeted room is best, maybe a part of the kitchen, hall, or bathroom. Then you can watch where it likes to poop and put a box in that spot if possible. It is usually easier to move the litter box to the area the rabbit prefers to use rather than trying to get him to use your specified area. The first time or two outside the cage the rabbit will most likely leave a few "pills" to mark where they have been. This is normal and will most likely stop after the first few times. For the litter boxes, buy large cat litter boxes or plastic dog baskets, or simply adapt plastic storage boxes. The key is for the litter trays to have high sides but easy access. Bunnies like to use the corner of the box but corner pans from pet stores are often too small. You can often create a taller corner (90° angle 7” high) out of various products (plastic file folder, tin foil wrapped cardboard, small sheet of metal, etc.) and place it just inside the litterbox in the rabbit’s favorite corner. Fill the boxes with a few inches of non-toxic litter such as the commercial paper-based liters (for example, Yesterday's News). Compressed pellets such as those used in pellet wood stoves are another economical option. Be sure not to use pine or cedar wood shavings or clumping cat litter, as all can harm your rabbit and clump in their intestinal tract.
Put one litter box into the rabbit's cage or near its bed. If it doesn't instinctively go to the box, put a few pellets and some urine-soaked paper or bedding into the boxes so the rabbit will have scent clues. Add a little timothy hay to one side of the box as rabbits instinctively poop and eat at the same time; the hay will help to entice the bunny into the box.
Clean up any urine or feces that are on the floor with diluted white vinegar as soon as possible so that the scent is gone and the rabbit isn't confused. If the bunny goes into the litter box, give it praise and a little treat. The more your rabbit associates the litter box with good feelings, the more it will want to spend time in the box and mark it with urine and feces. If the rabbit likes to lie in its box, that is a good thing, just keep it clean as sitting on soiled bedding over a long period of time can scald the bunny’s skin.
Whatever you do, be sure that the litter box is a happy time for your rabbit. Don't grab your bunny and put it into the box if it doesn't like to be handled, and don't scold your bunny if it messes on the floor. Rabbits are prey species and behave differently than cats and dogs. You don’t want to create a situation where the rabbit is afraid of the litterbox or even of you.
Remember that rabbits are creatures of habit, so once they get used to what's supposed to happen, they'll be more likely to using the litter box. Once your rabbit is using its box consistently, you can give it more freedom in the house if you want. Just be sure to watch it, and if you see the training starting to fail, put it back into the small area for more practice. If a bunny has quit using the litterbox, and nothing in your rabbit’s life has changed you should check with your vet to consider a health-related cause.