The present paper examines the calculation of thermal conductivity of insulating building materials made from plant particles. To determine the more suitable structure and particle size distribution of raw material to optimize the volume proportion between plant particles, binder and air, a tool for calculating the effective thermal conductivity of heterogeneous materials has been developed. The approach is both based on (i) the discrete element method to generate the volume element and (ii) the finite element method to calculate the homogenized properties. A 3D optical scanner has been used to record plant particle shapes and convert some of them into a cluster of discrete elements. These aggregates are initially randomly distributed in the numerical simulation but without any overlap, and then fall down into a container due to gravity and collide with neighboring particles according to a velocity Verlet algorithm. Once the RVE is built, the geometry is exported in the open-source Salome–Meca platform to be meshed. The calculation of the effective thermal conductivity of the heterogeneous volume is then performed using a homogenization technique, based on an energy method. The numerical model has been applied to packed beds of sunflower pith aggregates. The thermal conductivity of pith particles has been first measured to supply reliable data input for the numerical model. The remarkably low value of this material makes sunflower pith packed beds an excellent candidate for thermal insulation in the building industry. However, due to its low packing density, conduction and radiation heat transfer are significant. The first numerical simulations, considering only conductive heat transfer, differ thus from the measured thermal conductivity on packed beds. Finally, a more robust model taking into account the radiation contribution has been satisfactorily compared with the experimental results.