Abstract: Simulation-based inference (SBI) is a promising approach to leverage high fidelity cosmological simulations and extract information from the non-Gaussian, non-linear scales that cannot be modeled analytically. However, scaling SBI to the next generation of cosmological surveys faces the computational challenge of requiring a large number of accurate simulations over a wide range of cosmologies, while simultaneously encompassing large cosmological volumes at high resolution. This challenge can potentially be mitigated by balancing the accuracy and computational cost for different component models of the simulations while ensuring robust inference. To guide our steps in this, we perform a sensitivity analysis of SBI for galaxy clustering on various main components of the cosmological simulations: gravity model, halo-finder and the galaxy-halo distribution models. We infer two main cosmological parameters using galaxy power spectrum multipoles (two-point statistics) and the bispectrum monopole (three-point statistics) assuming a galaxy number density expected from current generation of galaxy surveys. We find that SBI is insensitive to changing gravity model between accureate and slow $N$-body simulations and approximate and fast particle mesh simulations. However, changing the methodology of finding the collapsed dark matter structures called halos which galaxies populate can lead to biased cosmological inferences. For models of how galaxies populate these halos, training SBI on more complex model leads to consistent inference for less complex models, but SBI trained on simpler models fails when applied to analyze data from a more complex model.