A hard and soft grid is two ways to simplify work with vertical spacing. A hard grid means that the height and spacing of the elements are always multiples of a grid unit. A soft grid does not limit the height of the elements in any way but only defines multiples of the spacing between them.
The soft grid is preferred in the design of sites and web applications. It gives more freedom in making design decisions and does not impose such restrictions as the hard grid. Any grid is just a tool. Only the project needs matter, not the limitations of the tool.
The spacing between design elements, which is proportional, multiples of the chosen unit, and a variety of vertical rhythms, will help achieve a neat, balanced look. So you don't need any hard grid or text baseline design restrictions. However, it may seem a simpler and more straightforward solution at first glance, especially technically.
There are several types of grids: column, modular, hierarchical, and others. Each type of grid can solve a particular problem. It all depends on the content and design requirements.
It is quite normal to combine different types of grids in one layout if the project's specifics require it. It all depends on the requirements. Grids only help to build the design neatly and uniformly.
There is no need to fit the sidebar into the page grid. The sidebar may well be separate from the overall grid, or it may have its own grid.
Often the sidebar has a fixed pixel width, and the content area stretches according to the width of the page. In that case, it isn't easy to fit the sidebar into the grid. And there is no point in doing that.
Also, the sidebar usually contains data that does not need to be divided into columns. Therefore, general grid columns are not required to build content in it. If the sidebar has data that needs columns, you can apply a unique grid to it, with the parameters you need to implement columns.
There is no need to fit the header into a single grid. First, the header is usually a line, not a column module, and the default line does not require a grid. Second, it isn't easy because the header can have a wide variety of elements, menus, and logos that you can't fit into columns.
So unless the content requires a direct relationship between the page columns and the header, there's no point in fitting it into the overall page grid. You can do without the grid in the header or make your own grid for it. The same goes for the footer.
There is no point in using a 12-column grid if three columns or any other number of columns is right for the content. The 12-column grid is often thoughtlessly used simply by default without any real relation to the project's requirements.
The grid is just a tool to help build the design. This tool should not dictate its own rules and set limits. The content and requirements of the project are what come first, and that's what matters. So it's okay if different screens or pages can have different grids with varying numbers of columns, depending on what content is on the page and what requirements need to be met.
It's perfectly normal to combine several grids in one layout, on one page, in one application. If the content requires different approaches, that's fine and allows you to build the most effective layout possible.
You shouldn't always fit content into one grid. It's not the grid that dictates the rules, and it's the content.
However, you should not overdo it. The grid is a tool that helps to achieve a neat design and uniformity of elements on all screens and pages of the project. So if you use two or three variants of the grid, it's probably enough. A mess of a bunch of different grid parameters will only lead to problems and a lack of unity in the design.
The grid is just a tool for organizing content. It's one way to build interfaces more efficiently, especially multi-page UI. The grid gives format and structure to speed up and simplify content.
If the content doesn't need to be grid-based, you shouldn't artificially create limits that don't make sense.
In general, you should not start working on a project with creating a grid. First, the content, its structure, and only then by the form, it makes sense to create a grid for the page or project as a whole.