System identification is the art of constructing mathematical models from observed data. It is a well established field with a history of over 40 years, characterized by a rich theoretical background, while it has proven its worth in many real life applications. On the other hand, Gaussian processes form a specific category of kernel based machine learning algorithms. Both methods aim at making predictions based on the past data. However, in contrast with the system identification algorithms, Gaussian processes do not deliver a parametric model but a mere (non-parametric) relation between the available inputs and outputs. Over the past few years the possible synergy between the two fields is extensively investigated. More specifically, the incorporation of the Gaussian process framework in the \ac{PEI} methods for \ac{LTI} systems was recently achieved. In this way, desirable properties of the Gaussian processes such as the increased flexibility and the minimum variance property of the estimator were included in the PEI framework. Moreover, these methods manage to incorporate simple prior knowledge to the algorithm through the sophisticated determination of the covariance (kernel) properties of the related coefficients. This synergy was also recently extended to the \ac{SID} framework for LTI systems. This is exactly the starting point of this thesis. After this point, we analyse the effect of various aspects on this new algorithm, such as the kernel structure, the effect of \ac{SNR} ratio and the effect of the available data points. More importantly, the effect of the past window value is extensively investigated, since its value is critical towards the accurate identification in the classical SID methods. In this thesis it is shown that the kernel based SID methods exhibit a superior accuracy compared to the up-to-date SID algorithms. Moreover, it is shown that they are the least affected by the choice of the past window value, thus opening the way for more automatic methods, less affected by the specific choices of the users. Following the examination of the LTI case, the possible synergy between Gaussian processes and SID methods for \ac{LPV} systems is under consideration. To this end, we start with an analytic investigation of the LPV SID methods in order to highlight their characteristics. At this point it becomes obvious that a direct use of the kernel methods for LTI systems is impossible due to the differences between the two classes of systems. Therefore, new methods were sought, opening the way to two novel approaches for the kernel based SID of LPV systems. The first one is based on the introduction of a prior on the LPV equivalent Markov parameters, while the second one introduces a prior on the time varying impulse response coefficients. The theoretical aspects of the proposed algorithms are then highlighted to reveal their merits and deficiencies. Moreover, the structure of the kernels is a crucial aspect of the kernel based SID methods for LPV systems. Especially for the second proposed approach, the proposed kernels balance between two desirable but contradictory characteristics. On the one hand, simple structures alleviate the computational burden of the involved (non-convex) optimization algorithms but they can be too restrictive and so they may fail to capture the underlying dynamics. On the other hand, rich kernel structures are expected to offer better results but only if they manage to avoid local-minima, while the computational time is expected to be a serious limitation. Our solution follows after an assiduous investigation of the coefficients to be estimated and the subsequent establishment of a correlation between the kernel structure and the impulse response coefficients. This could be seen as the LPV equivalent of introducing simple prior knowledge in the proposed methods. Finally, the validity of the proposed algorithms is verified through a series of identification examples. The main result of this thesis project is that the new, kernel based algorithms show a superior accuracy compared to the standard SID methods for LPV systems. From these algorithms, the so called ``LPV-RKHS-PBSID$_{opt}$'' algorithm exhibits the most accurate results, as it is both theoretically justified and also observed in the simulation examples. All in all, the performance of the kernel based SID methods for LPV systems shows a high potential, which can lead to a change of paradigm of how mathematical models can be constructed from the observed data.