What you need to do is get the scythe and then go around the back of the factory and to the right. Over there you can get the axe, and then you're able to enter the factory through the right-hand side of the building. You can chop down the mushroom trees leading up to a path where there are windows into the factory. Those little vents on the top of building roofs mark where there are entrances on the sides (though those are comparably rare).
Hope that helps!