Thin film nanocomposite (TFN) membranes contain nanoparticles in the thin polyamide (PA) top layer, resulting in a remarkable increase in water permeability without compromising the salt rejection. Mesoporous silica nanoparticles (MSN) have gained much attention as nanofillers for improved performance PA TFN membranes. However, aggregation of MSN inside the PA layer and their tendency to fast dissolution in aqueous solutions are serious challenges which strongly affect the separation performance of MSN-based TFN membranes. In this work, these challenges were addressed by controlling the functionalization of MSN with hydrophobic organo-silane, such as octadecyltrichlorosilane (OTS) or methyltrichlorosilane (MTS), before incorporating in the PA layer. MSN were synthesized first by sol-gel process and then functionalized using post-grafting method during silanization reaction. The functionalization was tuned in order to allow the hydrophobic alkyl group of the silane molecule to graft both the external surface of MSN as well as their interior pores or to modify only their external surface depending on the functionalization procedure and the silane concentration. The model of functionalization and the amount of OTS or MTS grafted on the surface were estimated through the nitrogen adsorption measurement and thermogravimetric analysis. The functionalized MSN with a particle diameter of ≈ 80 nm were thereafter easily dispersed in the organic solvent during the TFN membrane preparation via interfacial polymerization method. The membrane performance was then assessed based on water permeability and salt rejection measurements. Several parameters were found to have a strong influence on the membrane performance such as concentration of grafted OTS or MTS and the NPs loading inside the PA layer. The low aggregation and good integration of the functionalized nanofillers inside the PA layer produced TFN membranes with superior initial water permeability. The results revealed that the initial water permeability of the TFN membranes with OTS functionalized MSN achieved ≈ 65% higher initial permeability than the reference TFC membrane at the optimum OTS amount and NPs loading, without sacrificing the membrane selectivity. Whereas the corresponding TFN membranes with MTS functionalized MSN achieved ≈ 130% higher permeance but with 2% less salt rejection than the reference TFC membrane at the same conditions. The controlled functionalization of MSN nanofillers not only can improve membrane performance but also can provide a deeper understanding of the role of porous structure of MSN on the separation mechanism. This work clearly emphasizes the direct relationship between the internal pores of MSN inside the PA barrier layer and increasing or decreasing the water permeability of resulting TFN membranes. Furthermore, it was investigated, how hydrophobic functionalization of MSN can improve the stability of MSN in aqueous solutions and improve the stability of the respective TFN membranes over prolonged filtration time at different pH values. The results showed that TFN membranes containing the OTS-functionalized MSN had only ≈ 6% (or 7.5% for MTS-functionalized MSN) decline in salt rejection compared to 34.5% decline for the membrane containing unfunctionalized nanofillers accompanied by increasing in water permeability after 240 h filtration time (120 h at pH 5 then 120 h at pH 9). According to these results, it was concluded that the functionalized MSN have low dissolution tendency due to the formation of a protective organic layer leading to enhanced long-time stability of the TFN membranes. Finally, the durability of the barrier layer after a prolonged use for desalination performance of PA TFN membranes was also investigated.