A simple C++ module system with CMake

Unknown — Wed, 27 Nov 2024 00:00:00 +0000

One way to implement a simple module/plugin system in C++ using CMake.

By module or plugin, I mean a replaceable software components, where each serves a similar function, but with a different implementation. Not C++ 20 modules</a>.

Background</h2>
In my current role, my team and I have been in the gradual process of converting a lot of Python code into C++. The software handles large volumes of time-series datasets, and is consequently difficult to parallelise. With the volume of data increasing over time, the processing time of the Python processors had been steadily creeping up - taking on the order of hours for some datasets.
After converting legacy Python code for a dataset into C++, we typically see processing times drop by a factor of 10 or so - the conversions have been a great success. But as more code was implemented in the C++ application, build times have also steadily increased, to the point of slowing down day-to-day development and CI.
The original architecture</h2>
At a high level, the high-level purpose of this application is pretty simple: given a particular dataset, run the code which processes that dataset to generate our desired output. The single executable therefore contains all the code needed to handle all datasets.
In `main()</code>, we had something akin to the following:`
#include "dataset1.hpp" #include "dataset2.hpp" #include "dataset3.hpp" // ... void run_for_dataset(const std::string& dataset_name, ...) { if (dataset_name == "dataset1") { run_dataset_1(); } else if (dataset_name == "dataset2") { run_dataset_2(); } else if (dataset_name == "dataset3") { run_dataset_3(); } // ... throw std::runtime_error{"Unknown dataset"}; } int main(int argc, char argv[]) { run_for_dataset(...); } </code></pre> Under the hood, each dataset processor is implemented using a heavily-templated class. Each class inherits a thin virtual</code> base class, so we have a shared processing loop which uses dynamic dispatch on the processor object. Templated classes in are typically defined and implemented in headers, so that's what developers tended to stick to. This resulted in tons of template instantiations within a single translation: the main source file. Not only did this take a long time, consume gigabytes of memory (to the point where our build agents started failing), it's also wasteful for an individual developer's purposes. While developing for a specific dataset, one is typically only changing the implementation of and running on a single dataset. But each time the main executable is built for testing, there's a lot of recompiling and linking of irrelevant code for other datasets. At some point, developers started locally commenting out the code for other datasets in main()</code> to dramatically reduce their build times. But local patches are inherently fragile: producing merge conflicts, and introducing the risk that they are accidentally committed and included in a code review.
`It was clear something had to be done. While I suggested moving template implementations into source files as much as possible, this wouldn't help the extreme link times, and was very tedious to do manually. While alternative linkers likemold</a> are available, I haven't found it much help in the build time department. Ideally, we'd have a system that would allow building the implementation(s) for just the required dataset(s).`
`Other developers had started using a script which automatically edited main()</code> for you, while I started on a build system solution. This ended up taking much longer to merge than I expected.`
A simple run-time module system</h2> To summarise, I was aiming for was a solution which: completely avoided compiling & linking code for unwanted datasets</li> didn't require any local code changes to select the desired datasets</li> supported building shared libraries rather than just static, in case that helped build times</li> was reasonably straightforward to use</li> </ol> CMake is our build system, so checking number 4) may be inherently difficult to achieve. But my first thought was using CMake build options to pick which dataset targets the final executable should depend on. This would trivially check the first box, as a CMake build tool will only build and link targets which are required by the target being built. This should also make 3) easy enough - we should be able to trivially swap the module targets from STATIC</code> to SHARED</code> libraries. The question then is how to solve 2). Obviously the big if</code> block had to go, because we don't want to #include</code> the code for each dataset, and not all of them will be available. What we needed was a sort of plugin system, where each plugin would automatically register itself when it was compiled as part of the application. I struggle to recall exactly where I got this pattern from, but in the end I opted for a system which made use of a static initialisation: Create a static global map</li> Create a class which, in its constructor, inserts into that static global map information about a specific "module" which handles a dataset</li> Create a static instance of the above class for each module, particularly including a function for initialising the module</li> </ol> So by the time we hit main()</code>, all of the static objects for each module should be initialised, populating the global map. To run a specific module, we search that map for the name we're looking for, and execute the function to initialise and/or run it. I've implemented a simple cut-down version of this system in this Git repository</a>. Hopefully the code should be mostly self-explanatory, but I'll step through it below. Building & running the example project</h3> $ ./build/src/program Usage: ./build/src/program <module name> Available modules: module1 Module 1 does some stuff module2 Module 2 does some other stuff $ ./build/src/program module1 Module 1 has done some stuff $ ./build/src/program module2 Module 2 has done some stuff </code></pre> Build with all modules: $ cmake -B build -DENABLED_MODULES= >/dev/null \ && cmake --build build \ && ./build/src/program ninja: no work to do. Usage: ./build/src/program <module name> Available modules: module1 Module 1 does some stuff module2 Module 2 does some other stuff </code></pre> Enable just module1</code>: $ cmake -B build -DENABLED_MODULES=module1 >/dev/null \ && cmake --build build \ && ./build/src/program [2/2] Linking CXX executable src/program Usage: ./build/src/program <module name> Available modules: module1 Module 1 does some stuff </code></pre> Enable both modules: $ cmake -B build -DENABLED_MODULES='module1;module2' >/dev/null \ && cmake --build build \ && ./build/src/program [2/2] Linking CXX executable src/program Usage: ./build/src/program <module name> Available modules: module1 Module 1 does some stuff module2 Module 2 does some other stuff </code></pre> Trying to enable a nonexistent module: $ cmake -B build -DENABLED_MODULES=invalidmodule >/dev/null \ && cmake --build build \ && ./build/src/program CMake Error at src/CMakeLists.txt:26 (message): ENABLED_MODULES contains unknown module 'invalidmodule'. Full list: module1;module2 </code></pre> Building shared library modules with BUILD_SHARED_LIBS</code></a>: $ cmake -B build -DENABLED_MODULES= -DBUILD_SHARED_LIBS=ON >/dev/null \ && cmake --build build \ && ./build/src/program ninja: no work to do. Usage: ./build/src/program <module name> Available modules: module1 Module 1 does some stuff module2 Module 2 does some other stuff </code></pre> Module registration</h3> The secret sauce is in the module_registration</code> library</a>. module_registration.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> #include <string> </td></tr> 4</td> #include <map> </td></tr> 5</td> #include <memory> </td></tr> 6</td> #include <functional> </td></tr> 7</td> </td></tr> 8</td> /* </td></tr> 9</td> Base class for all modules. </td></tr> 10</td> / </td></tr> 11</td> struct BaseModule { </td></tr> 12</td> BaseModule(std::string parameter) : parameter_{std::move(parameter)} {}; </td></tr> 13</td> virtual ~BaseModule() = default; </td></tr> 14</td> </td></tr> 15</td> virtual int process() = 0; </td></tr> 16</td> </td></tr> 17</td> private: </td></tr> 18</td> std::string parameter_{}; </td></tr> 19</td> }; </td></tr> 20</td> </td></tr> 21</td> / </td></tr> 22</td> Stuff about a module, including how to initialise it. </td></tr> 23</td> / </td></tr> 24</td> struct ModuleInfo { </td></tr> 25</td> using Factory = std::function<std::unique_ptr<BaseModule>(std::string)>; </td></tr> 26</td> </td></tr> 27</td> std::string name; </td></tr> 28</td> std::string description; </td></tr> 29</td> Factory factory; </td></tr> 30</td> }; </td></tr> 31</td> </td></tr> 32</td> using ModuleRegistry = std::map<std::string, ModuleInfo, std::less<>>; </td></tr> 33</td> </td></tr> 34</td> / </td></tr> 35</td> Add a module to the global registry. </td></tr> 36</td> / </td></tr> 37</td> void register_module(ModuleInfo&& meta); </td></tr> 38</td> </td></tr> 39</td> </td></tr> 40</td> / </td></tr> 41</td> Get all available modules </td></tr> 42</td> / </td></tr> 43</td> auto get_modules() -> const ModuleRegistry&; </td></tr> 44</td> </td></tr> 45</td> / </td></tr> 46</td> Utility class for registering a module at static initialisation. </td></tr> 47</td> </td></tr> 48</td> C++ doesn't (yet) natively support running arbitrary code at static </td></tr> 49</td> initialisation, so we use a class constructor as a simple workaround. </td></tr> 50</td> / </td></tr> 51</td> struct RegisterModule { </td></tr> 52</td> RegisterModule(ModuleInfo info) { </td></tr> 53</td> register_module(std::move(info)); </td></tr> 54</td> } </td></tr> 55</td> }; </td></tr></tbody></table></code></pre> </details> In the header</a> we provide BaseModule</code>, which as the name suggests, is a base class for all modules. After it, we define a ModuleInfo</code> which contains some meta information for a module - in particular, how to initialise it in order to return a BaseModule&</code>. Of course, the factory</code> could be replaced by a function which actually goes off and does the processing, but this system is a bit closer to our architecture. It also means that we can run a lot of the common stuff separately, and call into the module primarily for its main processing loop. We then provide a couple of methods for accessing the global map - from here one referred to as the "registry". register_module()</code> adds a module to the registry. get_modules()</code> returns a constant reference to the registry. module_registration.cpp</a></summary> 1</td> #include "module_registration.hpp" </td></tr> 2</td> </td></tr> 3</td> #include <cstdio> </td></tr> 4</td> #include <fmt/core.h> </td></tr> 5</td> #include <fmt/format.h> </td></tr> 6</td> </td></tr> 7</td> namespace { </td></tr> 8</td> </td></tr> 9</td> // Use a static within a function as the global registry, to ensure it is </td></tr> 10</td> // always initialised before use. </td></tr> 11</td> auto get_registry() -> ModuleRegistry& { </td></tr> 12</td> static auto registry = ModuleRegistry{}; </td></tr> 13</td> return registry; </td></tr> 14</td> } </td></tr> 15</td> </td></tr> 16</td> } </td></tr> 17</td> </td></tr> 18</td> void register_module(ModuleInfo&& info) { </td></tr> 19</td> const auto [it, inserted] = get_registry().try_emplace(info.name, info); </td></tr> 20</td> if (not inserted) { </td></tr> 21</td> fmt::println(stderr, "Warning: duplicate module registered: {}", it->first); </td></tr> 22</td> } </td></tr> 23</td> } </td></tr> 24</td> </td></tr> 25</td> auto get_modules() -> const ModuleRegistry & { </td></tr> 26</td> return get_registry(); </td></tr> 27</td> } </td></tr></tbody></table></code></pre> </details> In the source file, we primarily have our global map, which is returned by get_registry()</code>. Notably, making it a static variable inside a function ensures it's always initialised before use. Implementing a module</h3> module1.hpp</a></summary> 1</td> #include "module_registration/module_registration.hpp" </td></tr> 2</td> </td></tr> 3</td> #include <fmt/format.h> </td></tr> 4</td> </td></tr> 5</td> namespace module1 { </td></tr> 6</td> </td></tr> 7</td> /* </td></tr> 8</td> Module 1 does some stuff. </td></tr> 9</td> / </td></tr> 10</td> class Module1 : public BaseModule { </td></tr> 11</td> public: </td></tr> 12</td> Module1(std::string parameter) : BaseModule{std::move(parameter)} {} </td></tr> 13</td> </td></tr> 14</td> int process() override { </td></tr> 15</td> fmt::println("Module 1 has done some stuff"); </td></tr> 16</td> return 1; </td></tr> 17</td> } </td></tr> 18</td> }; </td></tr> 19</td> </td></tr> 20</td> } // namespace module1 </td></tr></tbody></table></code></pre> </details> Lets say we have a module which is implemented like so. Within the source file, we register it with a static RegisterModule</code> object: module1_main.cpp</a></summary> 1</td> #include <fmt/format.h> </td></tr> 2</td> </td></tr> 3</td> #include "module1/module1.hpp" </td></tr> 4</td> #include "module_registration/module_registration.hpp" </td></tr> 5</td> </td></tr> 6</td> namespace module1 { </td></tr> 7</td> </td></tr> 8</td> [[maybe_unused]] static auto registrar_ = RegisterModule(ModuleInfo{ </td></tr> 9</td> .name = "module1", </td></tr> 10</td> .description = "Module 1 does some stuff", </td></tr> 11</td> .factory = [](std::string parameter) -> std::unique_ptr<BaseModule> { </td></tr> 12</td> return std::make_unique<Module1>(std::move(parameter)); </td></tr> 13</td> }, </td></tr> 14</td> }); </td></tr> 15</td> </td></tr> 16</td> } // namespace module1 </td></tr></tbody></table></code></pre> </details> The name of the variable doesn't matter of course. I've set [[maybe_unused]]</code> to prevent static analysers complaining about the used symbol - they can't determine its purpose, after all. main</h3> Finally, we come to main()</code>. Its purpose is to initialise and run the right module. I've implemented some basic command-line handling to make the program basically functional. main.cpp</a></summary> 1</td> #include <cstdlib> </td></tr> 2</td> #include <fmt/core.h> </td></tr> 3</td> #include <fmt/format.h> </td></tr> 4</td> #include <fmt/ranges.h> </td></tr> 5</td> </td></tr> 6</td> #include "module_registration/module_registration.hpp" </td></tr> 7</td> </td></tr> 8</td> int main(int argc, char argv[]) { </td></tr> 9</td> if (argc < 2) { </td></tr> 10</td> fmt::println("Usage: {} <module name>", argv[0]); </td></tr> 11</td> fmt::println(""); </td></tr> 12</td> fmt::println("Available modules:"); </td></tr> 13</td> for (const auto &[name, module] : get_modules()) { </td></tr> 14</td> fmt::println(" {}", name); </td></tr> 15</td> fmt::println(" {}", module.description); </td></tr> 16</td> } </td></tr> 17</td> </td></tr> 18</td> return EXIT_SUCCESS; </td></tr> 19</td> } </td></tr> 20</td> </td></tr> 21</td> if (auto result = get_modules().find(argv[1]); </td></tr> 22</td> result != get_modules().end()) </td></tr> 23</td> { </td></tr> 24</td> auto instance = result->second.factory("Some parameter"); </td></tr> 25</td> return instance->process(); </td></tr> 26</td> } else { </td></tr> 27</td> fmt::println(stderr, "Unknown module '{}'", argv[1]); </td></tr> 28</td> return EXIT_FAILURE; </td></tr> 29</td> } </td></tr> 30</td> } </td></tr></tbody></table></code></pre> </details> In a fully-fledged application, the command-line handling would naturally be much more complex. In our application, for example, most of the command-line parameters are passed into the module's factory. Build configuration glue</h3> Here's a snippet of the fun bit: the CMake build configuration. CMakeLists.txt</a></summary> 1</td> set(CMAKE_CXX_STANDARD 20) </td></tr> 2</td> </td></tr> 3</td> include(${CMAKE_CURRENT_LIST_DIR}/cmake/add_module_test.cmake) </td></tr> 4</td> include(${CMAKE_CURRENT_LIST_DIR}/cmake/add_module.cmake) </td></tr> 5</td> </td></tr> 6</td> add_subdirectory(module_registration) </td></tr> 7</td> </td></tr> 8</td> add_subdirectory(module1) </td></tr> 9</td> add_subdirectory(module2) </td></tr> 10</td> </td></tr> 11</td> set(ALL_MODULES </td></tr> 12</td> module1 </td></tr> 13</td> module2 </td></tr> 14</td> ) </td></tr> 15</td> set(ENABLED_MODULES ${ALL_MODULES} CACHE STRING </td></tr> 16</td> "Semicolon-separated list of enabled modules. Possible values: ${ALL_MODULES}" </td></tr> 17</td> ) </td></tr> 18</td> </td></tr> 19</td> # If the user leaves the option blank, build everything </td></tr> 20</td> if (NOT ENABLED_MODULES) </td></tr> 21</td> set(ENABLED_MODULES ${ALL_MODULES}) </td></tr> 22</td> endif() </td></tr> 23</td> </td></tr> 24</td> foreach(module IN LISTS ENABLED_MODULES) </td></tr> 25</td> if (NOT module IN_LIST ALL_MODULES) </td></tr> 26</td> message(FATAL_ERROR "ENABLED_MODULES contains unknown module '${module}'. " </td></tr> 27</td> "Full list: ${ALL_MODULES}") </td></tr> 28</td> endif() </td></tr> 29</td> endforeach() </td></tr> 30</td> </td></tr> 48</td> add_executable(program </td></tr> 49</td> main.cpp </td></tr> 50</td> ) </td></tr> 51</td> </td></tr> 52</td> target_link_libraries(program </td></tr> 53</td> PRIVATE </td></tr> 54</td> ${ENABLED_MODULES} </td></tr> 55</td> fmt::fmt </td></tr> 56</td> module_registration </td></tr> 57</td> ) </td></tr></tbody></table></code></pre> </details> As initially described, it creates a build option ENABLED_MODULES</code> which lists the CMake targets (one for each module) which is linked to the main executable. The list is semicolon-separated because that's how CMake does things. Some extra code would allow more sensible comma or space separated values. I've implemented some basic validation to ensure the specified targets are valid. Weird things and lessons learnt</h2> If you browse the CMake configuration in the example project, you'll notice it's a bit more complex than I made it out to be in this article. There's also a couple of commented out workarounds for issues I encountered, at least the version of CMake we were using at the time. Keeping the static registration objects in the libraries</h3> After implementing the initial proof of concept, the first hurdle I ran into was a pretty confusing one. None of the modules were registering themselves at static init. The RegisterModule</code> constructor wasn't being called, and I couldn't set a breakpoint on the static registrar_</code> variable within the module itself. I eventually realised that the static variables weren't present in the final executable, or in the module library, which I could check with nm --demangle</code>. At that point, I was building each module as a static library. That's the default type if BUILD_SHARED_LIBS</code></a> is OFF</code>. This seems to be default behaviour of GCC and the GNU linker when they encounter static objects that are apparently unused. I found quite a few discussions on how to work around this, including Static Variables Initialization in a Static Library, Example</a> (C++ Stories, 2018)</li> How to force gcc to link an unused static library</a> (Stack Overflow, 2013)</li> prevent gcc from removing an unused variable</a> (Stack Overflow, 2015)</li> </ul> With some suggestions including: Adding the GCC attribute attribute((used))</code> to registrar_</code>. With this, registar_</code> is included in the module static library, but not in the final executable. </li> Linking the static library with the -whole-archive</code> option. </li> </ul> Messing with the link flags as in the latter solution isn't particularly easy in CMake. but thankfully Object libraries</a> are functionally equivalent. This has the added benefit of being portable, rather than relying on linker-specific options. That's the solution I ended up going with: compile the module libraries as OBJECT</code> libraries. I haven't encountered any such issue when the modules are built as shared libraries, so there's no workaround required when BUILD_SHARED_LIBS=ON</code>. add_module.cmake</a></summary> 1</td> if (BUILD_SHARED_LIBS) </td></tr> 2</td> # With shared objects, both modules and the registration library must be </td></tr> 3</td> # shared to ensure the same global registry is being mutated. </td></tr> 4</td> set(MODULE_LIBRARY_TYPE SHARED) </td></tr> 5</td> else() </td></tr> 6</td> # Otherwise, we build individual modules as OBJECT libraries, so that the </td></tr> 7</td> # apparently unused static RegisterModule symbol is not omitted. </td></tr> 8</td> set(MODULE_LIBRARY_TYPE OBJECT) </td></tr> 9</td> endif() </td></tr></tbody></table></code></pre> </details> Resolving code coverage metrics</h3> After the module system was functioning and applied to all of our existing modules, I noticed that code coverage metrics were significantly lower compared to the master branch. We collect these code metrics using Gcov</a>, and generate a nice report using gcovr</a>. I think this is a bug in gcov. I found this mailing list discussion which produced a patch: Should templates with multiple instantiations contribute to summaries in gcov?</a> In the meantime, I worked out that we could use an intermediate static library between the module being tested and the final test executable. This essentially undoes the above workaround to keep the "unused" static registration objects in the library. I guess that also means the template instantiations which are actually unused by the test executable linger around and affect the coverage statistics. This workaround appears to have no impact on metrics in the example project (tested with GCC 13), but I left it commented out in the add_module_test</code> function to show how it'd work: add_module_test.cmake</a></summary> 1</td> # params <module name> [source files, ...] </td></tr> 2</td> function(add_module_test module_target) </td></tr> 3</td> set(sources ${ARGV}) </td></tr> 4</td> list(REMOVE_AT sources 0) </td></tr> 5</td> </td></tr> 6</td> set(options --coverage) </td></tr> 7</td> </td></tr> 8</td> # The following workaround may be necessary if your code coverage metrics </td></tr> 9</td> # seem lower than expected. I think this is due to gcov not accounting for </td></tr> 10</td> # multiple template instantiations in its metrics: </td></tr> 11</td> # https://www.mail-archive.com/gcc%40gcc.gnu.org/msg98590.html </td></tr> 12</td> # Add an intermediate static (or shared) library to remove the superfluous </td></tr> 13</td> # symbols from the module library. </td></tr> 14</td> </td></tr> 15</td> # add_library(${module_target}_test_library) </td></tr> 16</td> # target_link_libraries(${module_target}_test_library PUBLIC ${module_target}) </td></tr> 17</td> </td></tr> 18</td> # target_compile_options(${module_target}_test_library PUBLIC ${options}) </td></tr> 19</td> # target_link_options(${module_target}_test_library PUBLIC ${options}) </td></tr> 20</td> # target_link_libraries(${module_target}_test </td></tr> 21</td> # ${module_target}_test_library </td></tr> 22</td> # ) </td></tr> 23</td> </td></tr> 24</td> add_executable(${module_target}_test ${sources}) </td></tr> 25</td> </td></tr> 26</td> target_link_libraries(${module_target}_test </td></tr> 27</td> GTest::gtest_main </td></tr> 28</td> ${module_target} </td></tr> 29</td> ) </td></tr> 30</td> </td></tr> 31</td> target_compile_options(${module_target}_test PUBLIC ${options}) </td></tr> 32</td> target_link_options(${module_target}_test PUBLIC ${options}) </td></tr> 33</td> </td></tr> 34</td> gtest_discover_tests(${module_target}_test) </td></tr> 35</td> endfunction() </td></tr></tbody></table></code></pre> </details> Shared libraries haven't helped build times</h3> Also after implementing and porting all the modules to the system, I did some crude benchmarking on my development laptop with time</code>. I tested the following combinations with both static and shared libraries. a clean build with all modules enabled</li> an incremental build with all modules enabled</li> a clean build with a single module enabled</li> an incremental build with a single module enabled</li> </ul> Unsurprisingly, the single module, and particularly incremental build, were much, much faster than when all modules are enabled. Surprisingly, building with static libraries was consistently a bit faster than shared libraries by a few seconds, for a ~500MB executable with debug symbols. In theory, linking statically should be slower due to more code being copied into the executable, plus with link-time optimisation. But I'm not an expert, and my measurements didn't break down the build into compile and link times. I'm sure this all varies heavily with the compiler and linker used anyway. A lesson in gradual migration</h3> It took a lot of time working on this on and off. From getting an initial proof of concept working, to converting a handful of modules to the new system, to getting the go ahead to convert all the existing modules. Then the final push, where we tested it thoroughly in a lower environment, then merged and got it deployed. We had no trouble with the deployment, and the new system remains ever since. Part of the long turnaround was from finding the time alongside all the work that needed to be done. In hindsight, there was no real reason we had to convert all the module in one go. Certainly we get the most benefit from having all modules immediately optional, but that meant maintaining a branch over a long period, requiring many painful rebases. A gradual implementation would have been quite simple: search for a module using the mechanism described in this article, and use it if found, otherwise</li> fall back to the big if</code> described at the start of the article</li> </ol> It would have served as an early proof of concept to convert a single module, test, and deploy it, then gradually extend it to each module. It may have taken longer to see build time improvements, but perhaps the effort could've been shared amongst the team. The Defra pet database system is broken Unknown — Sat, 28 Sep 2024 00:00:00 +0000 England legally requires you to microchip and register cats and dogs, but the law has created perfect conditions for regulatory capture. I currently keep two cats. Ollie is a middle-aged cat who I took in for a friend who couldn't keep him any longer. The other, Napoli, I adopted as a kitten - both his parents were strays, though the mother was adopted by a neighbour. Transferring keepership</h2> When I took in the first, I was given his microchip number along with his other paperwork. I was sent an email from Petlog.org.uk, telling me that they're a "DEFRA compliant database", and to sign up for a Petlog account to update my details. That same email says I'll have to pay either £6 or £19.95 for their basic or premium account options respectively. If you search for "uk pet database search", you get dozens of sites offering this lookup service. From there I learnt this DEFRA database thing is part of the legal requirements surrounding pet registration</a>. Essentially, if you live in England: you must microchip your cats and dogs</li> you must register that microchip on one of a list of approved databases</li> and you must keep that information up to date</li> </ul> That same page says You might be charged a fee for updating your pet's microchip information. </blockquote> Problem is, I'm a software engineer. I know a thing or two about databases, and especially that it should cost a lot less than £6 or £19.95 a year for the privilege of being able to update a database record. But there's more providers than just Petlog. So surely the Great Market has granted us some competitor which charges a reasonable fee for registration, and doesn't make you pay to update your details? Thankfully, yes. I found MyPet</a>, which charged me £1 to register my new cat, and promised unrestricted ability to update my personal details. Great! Problem solved, right? Nope. Now if you search his microchip number, you get hits on two databases - one for me on MyPet, the other being my friend on Petlog. I replied directly to the email from Petlog, telling them to delete Ollie's records since he's now registered on a new database, with zero response. Their contact page</a> only lists phone numbers, promising a webchat which doesn't materialise, even after disabling my adblocker entirely. Petlog's pricing</h2> Petlog initial email at least had the courtesy to tell include their pricing. The email links to this page</a> to buy their product and register my keepership. Naturally, the more expensive option is selected for me. Hilariously, if you select "Basic" on that page, it now costs £10. Apparently they've inflated their price by 66% since early 2023, when they sent me that email. If you're browsing their site as a prospective customer, the best pricing information I can find is here</a>, which only says to "Sign up to Premium today for £19.95". No indication of how much Basic costs at all. For those not familiar with .org.uk</code> domains, these are typically used by non-profits and charities, which makes it particularly egregious to see such predatory pricing and behaviour on this site. Registering my kitten</h2> After thinking I'd gotten wise to the tricks, when going to get my new kitten chipped, I explicitly asked them how the microchip registration would work. The receptionist told me I'd have to register it. Great, I thought, I'll just register him on MyPet. But then after picking him back up, I'm told I'll get an email. And so I did. From yet another database provider, this time Identibase. "Thank you for registering with identibase". Fuck. As a reminder, you chose to not activate your pet’s Digital ID. This means that your data will continue to remain accessible on your pet's microchip, but should you wish to change any information on your pet's Digital ID in the future, you will need to activate the ID with one of our protection plans </blockquote> Wonderful. Just like Petlog, Identibase requires you to pay if you want to update your database record. This time, when I went to register the new microchip on MyPet, it blocked due to the microchip already being registered on another database, Identibase. Apparently they've added some validation since I last tried this with Ollie. Seeing no other option at the time, I went through their signup process, with the mind to immediately transfer the record to MyPet. Should be free and easy, right? Wrong. I sent them an email to admin@identibase.co.uk</code>, requesting the transfer and giving the microchip and contact details. To their credit over Petlog, I at least got a response on this one. However, the next day, I received an email from PETtrac, yet another database provider, telling me Identibase have requested a transfer to them. If I'm feeling generous, I'd guess the support person from Identibase selected the wrong drop-down on a form. However, I got no further response on that ticket, despite informing them of the mistake. A good while later I try again, emailing admin@identibase.co.uk</code> with the same message. This time, again quite promptly, I get an email from MyPet! From Conor, the same guy when I registered Ollie. Maybe it's a one-man show over there. Unfortunately, Conor informs me that they get a lot of transfer requests, and since it's a time-consuming manual process, they're only accepting transfers for "MyPet Premium" customers. Wonderful. The email contained a coupon code for 50%, bringing the price down to £6/year. I believe Conor when he says he has to deal with a ton of these requests. I sympathise with the manual effort needed, especially so if this is a one-man show. If things had gone as planned, I would've taken my kitten home, entered his new microchip number and details into MyPet myself, and that would have been that. But no. Now I have to sign up for a 7-day trial of a yearly subscription I don't want or need, all to create a bloody database record. So I paid that £6. At least it went to a tiny company rather than one of the big ones with absurd subscription fees. Thankfully that's all sorted now. Given MyPet's premium £6/year subscription</a> didn't actually offer anything I was interested in, I don't intend to renew it. Next chance I got, I asked a Vet from my practice what happened. She explained Identibase provides the practice's microchips, and by default they register implanted chips with them out of convenience. She was sympathetic about it being a nuisance to unify all of your pet records onto a single database. It's not her fault of course, and I held back mentioning my contempt for Identibase and their fees. I don't believe the Vets get any kickbacks from Identibase, but I suppose it's possible that the price of the chips is somehow subsidised by Identibase's pet database business. Sounds like the practice would've had no problem with leaving the registration to me, which is how I understood it would work. In hindsight, either the receptionist meant I would have to register (on Identibase), or they weren't fully informed on how the process worked. Perhaps if I'd explicitly requested they leave it to me when organising the procedure, this whole mess would've been avoided. Identibase's prices</h2> So, why do I complain about Identibase so much? The signup is free, but in order to update your details (once again, a legal requirement), you'll have to pay for a subscription. If you already have an account, here's the page they direct you to https://www.identibase.co.uk/account/account-type. Naturally, you can't see their pricing unless you've already made an account, which requires going through their microchip registration process (or that being done for you, more on that later). So here's a screenshot. £10/year gives you the ability to update your details as often as you like for all your pets registered on their service. You also get access to a "Missing Pet Helpline", which sounds like a generic advice service, and a "keeper certificate", which I imagine is an autogenerated PDF you can print if you like. If you fork out £30/year, if your pet goes missing, they'll make a post on their social media for you, it sounds like they'll send out SMS messages to nearby members. There's also some generic-sounding advice material. Should you want a service which actually may help you recover a lost pet, you can pay a one-time £119 for a GPS tracker. This is apparently a cloud-based GPS tracker</a>, which currently promises no subscription fees. The tracker's shop is currently selling the tracker for £129 with a coupon code, so hey, at least your £30/year saves you that tenner. Where did the law go so wrong?</h2> So, what caused this whole mess? Obviously, if I didn't have to register a microchip, there wouldn't be any problem. But I do think that the microchip system is an obviously good idea. The problem is that chip only contains a number: your details have to live elsewhere. A database is the obvious solution, but who runs the database? In this age of neoliberalism, obviously a private company should run that database! With strict regulations on uptime and data integrity and so forth, of course. But that would make a monopoly, right? Even an idiot can recognise that Monopoly Bad, so let's open it up to more than one company, as long as they pass our standards. Boom, state-enforced oligopoly! Apparently 23 competitors aren't enough to prevent predatory pricing like I've seen. In this particular case, we have a legal requirement for keeping a microchip registered, and your details up to date</li> no regulations against Vets registering a microchip for you</li> no automated, standard process for transferring between databases</li> </ol> And you have a nasty recipe for an incredibly simple service being marked up incredibly high, and customer lock-in. All enabled by regulatory capture. In the end I'm complaining about not spending £10 a year, which (for me) isn't a lot of money. But it's the principle of thing - if lawmakers hadn't bodged this in the first place, a lot of headache would've been avoided. Cat pictures in return for your time</h2> A Bandcamp plugin for Grayjay Unknown — Wed, 08 May 2024 00:00:00 +0000 Grayjay</a> is a media app which supports multiple platforms, similar to NewPipe</a>. I've recently been trying it out in place of NewPipe, probably after one too many times where a Youtube update broke it. Note that Grayjay currently has some weird "temporary" license</a>, so treat it as "source-available" software at best. It doesn't qualify as Free and Open Source, as of writing. A platform I've sorely missed support for is Bandcamp</a>, where I buy most of my music. Thankfully Grayjay has a fairly straightforward plugin system, where you implement their API (in Javascript, blergh). So I did that. Here it is</a>. And here's a QR code for installing it. Or if Grayjay finally supports adding plugins by URL, give it this: https://git.sr.ht/~williamvds/bandcamp-grayjay-plugin/blob/master/BandcampConfig.json</a> Don't worry, it'll only update when the version in the config file is incremented. Status</h2> Currently pretty jank, but does the job. The home page shows the most recent "notable albums" as on the Bandcamp homepage. Known issues: Album size and track length isn't always provided</li> Accessing band pages with custom domains isn't supported</li> Tracks that can't be streamed aren't displayed at all</li> It needs "unrestricted" internet access, because Grayjay doesn't support wildcard subdomain access (band pages are like http://band.bandcamp.com</a>)</li> </ul> Linking in C & C++ Unknown — Thu, 04 Apr 2024 00:00:00 +0000 Every C & C++ beginner's tutorial will step you through source and header files, .c</code> and .h</code> respectively for C, and... well take your pick for C++: .cpp</code> and .hpp</code>, .cxx</code> and .hxx</code>, or even .cc</code> and .hh</code>. One of the build errors you'll encounter with these types of languages is the link error. Something like undefined reference to ...</code>. This tends to confound developers, who roughly understand this is due to some mistake with the compile options or build configuration. That missing link (pardon the pun) is the linker. I'd say that's due to a common gap in these beginner tutorials, particularly in explaining the role of the linker. Understanding linking also explains the need for duplicating function and class signatures in both source and header files. So if you've found yourself confused and frustrated over these errors, read on to master the linker once and for all. Building an executable</h2> The common understanding is that the build process for a compiled language goes something like this: Which is true, but as explained above, for C & C++ it's missing the key step of linking. The reality for a simple executable is a bit more like this: The compiler doesn't produce the executables itself: it relies on the linker as a secondary application to do so. Those .o</code> files the compiler produces are called object files. I'll explain what those are shortly. There are several types of objects files. You've probably come across at least the last one before: object files on Unix-y platforms: .o</code></li> on Windows; sometimes .obj</code></li> </ul> </li> static libraries on Unix-y platforms: .a</code> for Archive</li> on Windows: .lib</code></li> </ul> </li> shared libraries on Unix-y platforms: .so</code> for Shared Object</li> on Windows: .dll</code> for Dynamic Link Library</li> </ul> </li> </ul> These each serve slightly different purposes. The key difference is that the first two, object files and static libraries are build-time dependencies. i.e., once the executable has been built, they are no longer needed. You can remove them and the executable will still run. Shared libraries are run-time dependencies of the executable. i.e., once the executable has been built and linked to these libraries, they must (or sometimes less strongly, should) be available for the executable to run correctly. Git repository</h2> To dig in to how this all works, I've set up a very simple toy C++ project. You can find it in this repository</a>. I'll link to specific commits as we go along, but for convenience, some files and diffs will be embedded into this article. Objects</h2> See commit 5ec3b5f2 Initial example (diff</a>, tree</a>) All object files share the same purpose: they provide symbols. Generally, that includes: functions</li> data (variables, constants)</li> classes (which are a combination of both data and functions)</li> </ul> A particularly useful tool for digging into symbols in object files is nm</code>, which is provided by GNU binutils</a>. Note that when certain build configurations like Link-Time Optimisation (LTO) are enabled, you'll have to use nm</code> with compiler specific plugins. GCC provides a wrapper program, gcc-nm</code>. </aside> Here's the results of using it against a couple of object files generated from a very simple C++ source file. main.cpp</a></summary> 1</td> extern "C" { </td></tr> 2</td> </td></tr> 3</td> int square(int num) { </td></tr> 4</td> return num num; </td></tr> 5</td> } </td></tr> 6</td> </td></tr> 7</td> int operand = 5; </td></tr> 8</td> </td></tr> 9</td> } </td></tr> 10</td> </td></tr> 11</td> int main() { </td></tr> 12</td> return square(operand); </td></tr> 13</td> } </td></tr></tbody></table></code></pre> </details> The extern "C"</code> block tells the compiler to use C's rules for linking code within that block. I'll explain what those differences are later on. You'll often see C or C++ headers with the following pattern: #ifdef cplusplus extern "C" { #endif // function declarations and such #ifdef cplusplus } // extern "C" #endif </code></pre> The extern "C"</code> is conditionally wrapped around the declarations when using a C++ compiler, so C linking rules are always used. That's important if you're linking a C++ application to a C library. Let's run nm</code> against the generated object file: main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 000000000000000f T main 0000000000000000 D operand 0000000000000000 T square </code></pre> </details> For each symbol, nm</code> outputs <address> <type> <symbol name> </code></pre> address is the address in ?? type indicates both the purpose of the symbol (e.g. function, data), and which section it exists in the object. symbol name is, unsurprisingly, the name of the symbol. Notice how for our functions operand</code> and main</code>, the symbol name is simply the name of the function. So we've got an entry for the two functions in main.cpp</code> of type T</code>, and the variable with type d</code>. But what exactly do the types mean? Object layout and symbol types</h3> The nm</code> manual page explains the types of objects in "D" "d" The symbol is in the initialized data section. ... "T" "t" The symbol is in the text (code) section. </code></pre> So T</code> for functions because they live in the text section, and D</code> for variables because they live in the data section. As this suggests, an object file is split up into different sections for different types of symbols. Object formats vary by platform, GNU/Linux uses the Executable and Linking Format (ELF) originally from Unix System V1</a>. Checking the object files with file</code> shows: $ file ./build/CMakeFiles/example.dir/main.cpp.o ./build/CMakeFiles/example.dir/main.cpp.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped </code></pre> Note ELF prefixes section names with a period, so in that context, you'll usually see them referred to as .text</code> and .data</code>. The sections are just two among many in an ELF object, but they contain most of the symbols the average developer is interested in. Some other notable ones include: .strtab</code>: string tables, constant strings that are embedded in the program</li> .rodata</code>: read-only data. When symbols in this section are loaded into a process' memory, that memory section is typically set to non-writable.</li> .init</code> and .fini</code>: instructions used during a process' static init and destruction respectively.</li> .ctors</code> and .dtors</code>: pointers to C++ constructor and destructor functions respectively.</li> </ul> It probably won't surprise you that read-only data contains const</code> global variables, i.e. those which exist for the lifetime of the program and are immutable. As a quick demo, I'll make operand</code> const</code>: See commit edc20aec Make operand const (diff</a>, tree</a>) You'll see that operand</code> now has type r</code>, for read-only data: main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 000000000000000f T main 0000000000000000 T square 0000000000000000 r operand </code></pre> </details> Executables</h3> Use nm</code> on an executable and you'll see a lot more symbols which you likely won't recognise. Highlighted alongside these are our symbols from main.cpp.o</code>: executable symbols</a></summary> $ nm --demangle ./build/example 0000000000004010 B __bss_start w __cxa_finalize@GLIBC_2.2.5 0000000000004000 D __data_start 0000000000004000 W data_start 0000000000004008 D __dso_handle 0000000000003df0 d _DYNAMIC 0000000000004010 D _edata 0000000000004018 B _end 000000000000113c T _fini 0000000000003fe8 d _GLOBAL_OFFSET_TABLE_ w __gmon_start 0000000000002008 r GNU_EH_FRAME_HDR 0000000000001000 T _init 0000000000002000 R _IO_stdin_used w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U __libc_start_main@GLIBC_2.34 0000000000001128 T main 0000000000001119 T square 0000000000001020 T _start 0000000000004010 D __TMC_END 0000000000002004 r operand </code></pre> </details> You don't have to worry about the other symbols, they're prefixed with underscores to indicate they're private, internal implementation details. Most of these are internal to the GNU Compiler Collection (GCC), the compiler suite with which this executable was compiled. Definitions in header files</h2> Let's imagine square</code> and operand</code> are common parts of our program which will be reused in other source files. A common pattern for C & C++ programs to share code is using header files. So we simply copy the definitions into a header file, right? Let's give that a shot: See commit 58e3d1a0 Move common code into a header file (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> extern "C" { </td></tr> 4</td> </td></tr> 5</td> int square(int num) { </td></tr> 6</td> return num num; </td></tr> 7</td> } </td></tr> 8</td> </td></tr> 9</td> int operand = 5; </td></tr> 10</td> </td></tr> 11</td> } // extern "C" </td></tr></tbody></table></code></pre> </details> After doing so, our executable still builds and executes successfully: build log</a></summary> $ cmake -B build -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/avery/scratch/public/linking-by-example/build $ cmake --build build [1/2] Building CXX object CMakeFiles/example.dir/main.cpp.o [2/2] Linking CXX executable example </code></pre> </details> And the symbols are identical to before: executable symbols</a></summary> $ nm --demangle ./build/example 0000000000004010 B bss_start w __cxa_finalize@GLIBC_2.2.5 0000000000004000 D __data_start 0000000000004000 W data_start 0000000000004008 D __dso_handle 0000000000003df0 d _DYNAMIC 0000000000004010 D _edata 0000000000004018 B _end 000000000000113c T _fini 0000000000003fe8 d _GLOBAL_OFFSET_TABLE_ w __gmon_start 0000000000002008 r GNU_EH_FRAME_HDR 0000000000001000 T _init 0000000000002000 R _IO_stdin_used w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U __libc_start_main@GLIBC_2.34 0000000000001128 T main 0000000000001119 T square 0000000000001020 T _start 0000000000004010 D __TMC_END__ 0000000000002004 r operand </code></pre> </details> So job done, right? Not quite. Let's see what happens when we add another source file to our project. The One Definition Rule</h3> See commit 3eab681e Add another source file (diff</a>, tree</a>) This file also wants to reuse square</code>, so it includes the header file which defines the function. library.cpp</a></summary> 1</td> #include "library.hpp" </td></tr> 2</td> </td></tr> 3</td> int square_10() { </td></tr> 4</td> return square(10); </td></tr> 5</td> } </td></tr></tbody></table></code></pre> </details> Add this source file to the project and try compiling again, you'll hit a build error: build log</a></summary> $ cmake -B build -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/avery/scratch/public/linking-by-example/build $ cmake --build build [1/3] Building CXX object CMakeFiles/example.dir/main.cpp.o [2/3] Building CXX object CMakeFiles/example.dir/library.cpp.o [3/3] Linking CXX executable example FAILED: example : && /usr/bin/c++ CMakeFiles/example.dir/library.cpp.o CMakeFiles/example.dir/main.cpp.o -o example && : /usr/bin/ld: CMakeFiles/example.dir/main.cpp.o: in function `square': main.cpp:(.text+0x0): multiple definition of `square'; CMakeFiles/example.dir/library.cpp.o:library.cpp:(.text+0x0): first defined here /usr/bin/ld: CMakeFiles/example.dir/main.cpp.o:(.data+0x0): multiple definition of `operand'; CMakeFiles/example.dir/library.cpp.o:(.data+0x0): first defined here collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. </code></pre> </details> Now it's complaining about multiple definition of</code> both our function square</code> and our variable operand</code>. Notice the ld returned 1 exit status</code>, indicating that this error is actually produced by ld</code>, which is GCC's linker. So, the linker isn't happy that these symbols are defined multiple times. This is the linker's default behaviour, because the linker doesn't know what to make of the ambiguity. This is called the One Definition Rule (ODR) in C & C++2</a>. C++'s rule is slightly different, because it has additional language features which affect linkage. If we look into the object files you'll find that both main.cpp.o</code> and library.cpp.o</code> contain the symbols square</code> and operand</code>: main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 000000000000000f T main 0000000000000000 D operand 0000000000000000 T square </code></pre> </details> library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 D operand 0000000000000000 T square 000000000000000f T square_10() </code></pre> </details> So the compiler has generated the objects for each source file correctly, but linker gives up when trying to join these two object files together into the executable. While in this particular case, we know that both of these symbols were generated from the same code. In the case of square</code>, we know the symbols are compatible. But things get a bit messier with operand</code>, which at this point is once again a mutable global variable. But what if we or some third-party code had defined its own symbols also called square</code> and operand</code>? The linker has no way to determine that these and our own versions are different and probably incompatible. So the linker gives up, and informs you of the multiple definitions, leaving you to diagnose the issue. Remember, #include</code> is an instruction to include the full text of another file at that line, also including any transient #include</code>s in that file. So rather than just the source file' contents that is compiled, it's the source file's contents, plus any included files. This combination is generated by the preprocessor, the part of the compiler which handles all the #</code> instructions in C & C++ code. The output of the preprocessor is called a translation unit, and is then passed on to the compiler proper to generate code. If we visualise the preprocessor as its own separate stage, our project currently looks like this: Those .i</code> files are the translation units in between. While you can explicitly ask the compiler to run just the preprocessing stage, preprocessing and compilation normally occurs in a single step. The static</code> specifier</h3> So, each object has created its own copy of operand</code> and square</code>. How can we appease the linker? An easy workaround with C linkage is adding the static</code> keyword to the definitions. See commit 29ebb7ab Workaround duplicate symbols with static (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> extern "C" { </td></tr> 4</td> </td></tr> 5</td> static int square(int num) { </td></tr> 6</td> return num num; </td></tr> 7</td> } </td></tr> 8</td> </td></tr> 9</td> static int operand = 5; </td></tr> 10</td> </td></tr> 11</td> } // extern "C" </td></tr></tbody></table></code></pre> </details> Now the executable is successfully compiled: build log</a></summary> $ cmake -B build -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/avery/scratch/public/linking-by-example/build $ cmake --build build [1/3] Building CXX object CMakeFiles/example.dir/library.cpp.o [2/3] Building CXX object CMakeFiles/example.dir/main.cpp.o [3/3] Linking CXX executable example </code></pre> </details> Investigating the object files again, you'll find the symbols are still duplicated between library.cpp.o</code> and main.cpp.o</code>, but there's a subtle change in square</code> and operand</code>: the T</code> and D</code> are now respectively lowercased. main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 000000000000000f T main 0000000000000000 t square 0000000000000000 d operand </code></pre> </details> library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 t square 000000000000000f T square_10() 0000000000000000 d operand </code></pre> </details> nm</code> explains the distinction thus: If lowercase, the symbol is usually local; if uppercase, the symbol is global (external). </blockquote> A local/internal symbol in one object won't be linked across other objects. Conversely, a global/external symbol will be linked across objects. Also, notice that square</code> and operand</code> are both duplicated in the final executable: executable symbols</a></summary> $ nm --demangle ./build/example 0000000000001147 T main 0000000000001119 t square 0000000000001138 t square 0000000000004010 d operand 0000000000004014 d operand </code></pre> </details> When writing C, it's suggested to make any functions that are only used within a single source file static</code>. It's particularly important in C, because there's only one global namespace. That means it's easy for symbol names to collide across different translation units. </aside> Thinking about our particular use case - a function and variable in a header - it's worth considering whether static</code> gives you desirable behaviour. square</code> is compiled for each translation unit which includes library.hpp</code>. That can inflate build times due to the redundant compilation. It's not a big deal with just these two symbols here, but imagine a mature codebase where lots of code is defined as static</code> functions in headers. Similarly, the duplicate symbols can inflate the size of your executable. Things are more complicated for operand</code>, which is a mutable global. With a copy in each translation unit, if your program depends on mutating that global, duplicates can cause surprising behaviour. The inline</code> specifier</h3> An alternative to static</code> is the inline</code> keyword, which can serve a similar purpose, but affects how everything is linked together at the end. inline</code> was originally intended for telling the compiler you want a function's code to be inserted wherever it is called, avoiding the overhead of a regular function call. In order to be inlined, a function's definition must be available in every translation unit that calls the function, so definitions are stored in headers. To prevent this causing duplicate symbol errors, a function defined with inline</code> has different link rules. Modern compilers often ignore the inline</code> qualifier when generating code, instead deciding for itself when a function should be inlined. Naturally, higher optimisation levels introduce more inlining of code. Consequently, in modern C & C++ code, inline</code> is often used for its linkage properties rather than code generation properties. </aside> See commit 93725c2f Make header symbols inline (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> extern "C" { </td></tr> 4</td> </td></tr> 5</td> inline int square(int num) { </td></tr> 6</td> return num num; </td></tr> 7</td> } </td></tr> 8</td> </td></tr> 9</td> inline int operand = 5; </td></tr> 10</td> </td></tr> 11</td> } // extern "C" </td></tr></tbody></table></code></pre> </details> main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 0000000000000000 T main 0000000000000000 u operand 0000000000000000 W square </code></pre> </details> library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 W square 0000000000000000 T square_10() </code></pre> </details> executable symbols</a></summary> $ nm --demangle ./build/example 0000000000001138 T main 0000000000004010 u operand 0000000000001129 W square </code></pre> </details> There's only one copy of the symbols in the final executable, and square</code> is now a global weak symbol, while operand</code> is a global unique symbol. Here are the explanations from nm</code>: "u" The symbol is a unique global symbol. This is a GNU extension to the standard set of ELF symbol bindings. For such a symbol the dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use. ... "W" "w" The symbol is a weak symbol that has not been specifically tagged as a weak object symbol. When a weak defined symbol is linked with a normal defined symbol, the normal defined symbol is used with no error. When a weak undefined symbol is linked and the symbol is not defined, the value of the symbol is determined in a system-specific manner without error. On some systems, uppercase indicates that a default value has been specified. </blockquote> So a weak symbol is a kind of soft dependency, and linking won't fail if it's not present. Also, a symbol declared weak can be overridden by a strong symbol with the same name. Implicit here is that weak symbols don't collide with one another, and the linker will pick just one definition if there are several. In our case, the weak symbols are capitalised, meaning a default definition has been provided. So the end result is the same for both symbols: the linker picks one definition of each, and there's only one copy in the final executable. You may be aware that in C++ you can override the global operators new</code> and delete</code>. This is useful if, for example, you want to use your own allocator for dynamic memory allocations. On GNU/Linux, this is supported using weak symbols: the default operator definitions are declared as weak symbols, but your override is be a strong symbol. The linker will pick your strong override over the weak default definitions. </aside> The general suggestion in C++ is that any functions defined in headers should be marked inline</code>, to avoid running afoul of the One Definition Rule. C++ has a few more options, which may be more appropriate in certain contexts. C++'s constexpr</code> and consteval</code></h3> Newer versions of C++ add a few extra keywords which avoid creating multiple definitions. constexpr</code>3</a>, added in C++ 17, declares that a function or variable may be evaluated at compile-time. consteval</code>4</a>, added in C++ 23, declares that a function or variable must be evaluated at compile-time. Both of these imply inline</code>, so you can use them in place of inline</code> in headers. constexpr</code> and consteval</code> allow the compiler to eliminate some of your code by evaluating it at compile time, trading build time for runtime performance. A recommendation in C++ is to constexpr</code> everything that you can. Consider wisely what you consteval</code>, because not everything needs to be evaluated at compile-time. There are other cases where inline</code> is implicit, including: member function definitions that are inside a class definition</li> template function definitions and template variables</li> </ul> Sharing definitions with other source files</h2> Another common method for following the One Definition Rule is by moving definitions into source files, and marking declarations with extern</code>. This keyword indicates that a symbol has external linkage, so it's the opposite of static</code>, and these specifiers are mutually exclusive. extern</code> allows you to declare symbols which are defined in other translation units, leaving the linker to link references to those symbols later. Note that function declarations are implicitly extern</code>. Let's make square</code> and operand</code> extern, and move their definitions into library.cpp</code>: See commit 73d3221f Move definitions to source file (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> extern "C" { </td></tr> 4</td> </td></tr> 5</td> int square(int num); </td></tr> 6</td> </td></tr> 7</td> extern int operand; </td></tr> 8</td> </td></tr> 9</td> } // extern "C" </td></tr></tbody></table></code></pre> </details> library.cpp</a></summary> 1</td> #include "library.hpp" </td></tr> 2</td> </td></tr> 3</td> extern "C" { </td></tr> 4</td> </td></tr> 5</td> int square(int num) { </td></tr> 6</td> return num num; </td></tr> 7</td> } </td></tr> 8</td> </td></tr> 9</td> int operand = 5; </td></tr> 10</td> </td></tr> 11</td> } // extern "C" </td></tr> 12</td> </td></tr> 13</td> int square_10() { </td></tr> 14</td> return square(10); </td></tr> 15</td> } </td></tr></tbody></table></code></pre> </details> Looking into the objects that are created, we can see a new type of symbol has appeared in main.cpp.o</code>: main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 0000000000000000 T main U operand U square </code></pre> </details> Both operand</code> and square</code> are undefined symbols. These are essentially placeholders for the linker to resolve during linking. Unlike a weak symbol as mentioned earlier, if an undefined symbol is not present during linking, the linker raises an error. E.g., an executable which refers to an unlinked undefined function isn't valid, because the code for that function isn't available. Looking at library.cpp.o</code> and the final executable, square</code> and operand</code> are in the regular data and text sections: library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 D operand 0000000000000000 T square 000000000000000f T square_10() </code></pre> </details> executable symbols</a></summary> $ nm --demangle ./build/example 0000000000001138 T main 0000000000004010 D operand 0000000000001119 T square </code></pre> </details> This shows that the linker resolved the references to square</code> and operand</code> in main.cpp.o</code> to the definitions provided in library.cpp.o</code>. C++ specific stuff</h2> Linkage</h3> So, why are the link rules different between C and C++? Because the rules for C didn't have to account for C++ features like namespaces, overloading, and templates. Let's take our existing example and remove the enwrapping extern "C"</code> so that the C++ link rules apply. See commit b544fe3a Remove extern C (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> int square(int num); </td></tr> 4</td> </td></tr> 5</td> extern int operand; </td></tr></tbody></table></code></pre> </details> library.cpp</a></summary> 1</td> #include "library.hpp" </td></tr> 2</td> </td></tr> 3</td> int square(int num) { </td></tr> 4</td> return num num; </td></tr> 5</td> } </td></tr> 6</td> </td></tr> 7</td> int operand = 5; </td></tr> 8</td> </td></tr> 9</td> int square_10() { </td></tr> 10</td> return square(10); </td></tr> 11</td> } </td></tr></tbody></table></code></pre> </details> library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 D operand 0000000000000000 T square(int) 000000000000000f T square_10() </code></pre> </details> Now the symbol names look a bit more like function declarations, because that information is encoded in the symbol name. I say encoded, because within the object file, C++ symbol names are mangled into a different form. Since I passed the --demangle</code> option to nm</code>, it has conveniently converted the names back into a human-readable form. Without that option, things look a bit different: $ nm ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 D operand 0000000000000000 T _Z6squarei 000000000000000f T _Z9square_10v </code></pre> You can still roughly make out the names, but add namespaces and classes into your files and mangled symbol names become a lot more unwieldy. You may also notice a pattern, e.g. the i</code> suffix for square</code> indicates an int</code> parameter, and v</code> at the end of square_10</code> indicates void</code> since it takes no parameters. Mangling rules vary by compiler, so you may see a different pattern if you're not using GCC as I am. If you're curious, here's an example with namespaces and operator overloading: See commit 9e7dab9b C++ overloading example (diff</a>, tree</a>) And here's the compile error you get, should you try overloading with C linkage rules: See commit 39cfc50e C overloading attempt (diff</a>, tree</a>) Templates</h3> Commonly, C++ template definitions are stored in headers, and inline</code> by default. Thinking about common templates like those provided by the standard library, e.g. std::vector<T></code>, there can be hundreds of instantiations with different types for T</code> in a given program. Instantiation is when the templated code is actually generated on-demand during compilation, with the template arguments substituted. Let's convert square</code> to a template and see what happens: See commit 624f9f11 Replace with a C++ template (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> auto square(T num) { </td></tr> 5</td> return num num; </td></tr> 6</td> } </td></tr> 7</td> </td></tr> 8</td> extern int operand; </td></tr></tbody></table></code></pre> </details> library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 D operand 0000000000000000 W auto square<int>(int) 0000000000000000 T square_10() </code></pre> </details> main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 0000000000000000 T main U operand 0000000000000000 W auto square<long>(long) </code></pre> </details> executable symbols</a></summary> $ nm --demangle ./build/example 0000000000001129 W auto square<int>(int) 000000000000114e W auto square<long>(long) 0000000000001119 T square_10() </code></pre> </details> Now you can see two instantiations of square</code> in the final executable - one comes from library.cpp</code>, which uses square<int>()</code> in square_10()</code>, the other comes from main.cpp</code>, which uses square<long>()</code> in main()</code>. Template instantiation occurs in specific circumstances: When a class template specialization is referenced in context that requires a complete object type, or when a function template specialization is referenced in context that requires a function definition to exist 5</a> </blockquote> If you analyse C++ build times with tools like ClangBuildAnalyzer</a>, you'll often find a significant amount of time is spent instantiating templated code - particularly standard library classes, due to their popularity. One workaround is using precompiled headers, which as the name suggests, are headers compiled before the rest of your code. The idea is to compile frequently used code once and reuse it, rather than generating the same code repeatedly while compiling different translation units. When dealing with templated classes, you can use explicit template initialisation to force the compiler to generate some templated code. It looks like this: template <typename T> void Function(T t) { // ... } // Instantiate Function with template argument int template void Function<>(int); template <typename T> class Class { // ... }; // Instantiate Class and its methods with template argument int template class Class<int>; </code></pre> Try to not confuse the syntax with template specialisation, which generally looks like this: template <typename T> void Function(T t) { // ... } // Declare a specialised Function with template argument int template <> void Function<>(int); </code></pre> Oh the difference an empty pair of angled braces can make! Alternatively, if you know the set of template parameters that you need up front, you can do similar to regular functions and classes: declare them in headers, and define them in source files. One extra step is needed: adding template instantiations in the source file to generate the required code. In this case we know we need square instantiation with the template parameters int</code> and long</code>. So, let's move the definition for square</code> into library.cpp</code>, and leave a declaration in library.hpp</code>. See commit 343d4246 Explicit template instantiation (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> extern T square(T num); </td></tr> 5</td> </td></tr> 6</td> extern int operand; </td></tr></tbody></table></code></pre> </details> library.cpp</a></summary> 1</td> #include "library.hpp" </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> T square(T num) { </td></tr> 5</td> return num num; </td></tr> 6</td> } </td></tr> 7</td> </td></tr> 8</td> template int square<>(int); </td></tr> 9</td> template long square<>(long); </td></tr> 10</td> </td></tr> 11</td> int operand = 5; </td></tr> 12</td> </td></tr> 13</td> int square_10() { </td></tr> 14</td> return square(10); </td></tr> 15</td> } </td></tr></tbody></table></code></pre> </details> And looking at the object files, things look similar to the previous example where we declared square</code> in the header and defined it in the source file: main.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/main.cpp.o 0000000000000000 T main U operand U long square<long>(long) </code></pre> </details> library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/library.cpp.o 0000000000000000 D operand 0000000000000000 W int square<int>(int) 0000000000000000 W long square<long>(long) 0000000000000000 T square_10() </code></pre> </details> Instantiations in multiple translation units</h3> This all leads into my original reason for diving into this: a curious case where we hit linking issues relating to templates. This particular situation involved a templated class which was implemented in a source file. We wanted to create a subclass from it, but have the subclass reside in a different library separate to the superclass. What this boiled down to was needing multiple instantiations in different translation units: one for the parent class template, and one for the child class template. The following is a rough sketch of the code. See commit d6cb4e38 Templated subclass attempt (diff</a>, tree</a>) library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> class TemplatedClass </td></tr> 5</td> { </td></tr> 6</td> public: </td></tr> 7</td> virtual ~TemplatedClass() = default; </td></tr> 8</td> </td></tr> 9</td> virtual int method(); </td></tr> 10</td> }; </td></tr></tbody></table></code></pre> </details> library.cpp</a></summary> 1</td> #include "library.hpp" </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> int TemplatedClass<T>::method() { return 42; } </td></tr></tbody></table></code></pre> </details> other_library.hpp</a></summary> 1</td> #pragma once </td></tr> 2</td> </td></tr> 3</td> #include "library.hpp" </td></tr> 4</td> </td></tr> 5</td> template <typename T> </td></tr> 6</td> class ChildClass : public TemplatedClass<T> </td></tr> 7</td> { </td></tr> 8</td> public: </td></tr> 9</td> int method() override; </td></tr> 10</td> }; </td></tr></tbody></table></code></pre> </details> other_library.cpp</a></summary> 1</td> #include "other_library.hpp" </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> int ChildClass<T>::method() { return 1; } </td></tr> 5</td> </td></tr> 6</td> template class ChildClass<int>; </td></tr> 7</td> template class TemplatedClass<int>; </td></tr></tbody></table></code></pre> </details> CMakeLists.txt</a></summary> cmake_minimum_required(VERSION 3.27) project(example LANGUAGES CXX ) add_executable(example library.cpp other_library.cpp main.cpp ) </code></pre> </details> So we have a parent class defined in one source file, and a child class defined in its own source file. The source file for the child class also explicitly instantiates the template for both the child and parent class. Finally, I definitely added the new source file to the project. Despite all that, we get a missing symbol error for the parent class when building: build log</a></summary> $ cmake -B build -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/avery/scratch/public/linking-by-example/build $ cmake --build build [1/4] Building CXX object CMakeFiles/example.dir/library.cpp.o [2/4] Building CXX object CMakeFiles/example.dir/other_library.cpp.o [3/4] Building CXX object CMakeFiles/example.dir/main.cpp.o [4/4] Linking CXX executable example FAILED: example : && /usr/bin/c++ CMakeFiles/example.dir/library.cpp.o CMakeFiles/example.dir/other_library.cpp.o CMakeFiles/example.dir/main.cpp.o -o example && : /usr/bin/ld: CMakeFiles/example.dir/other_library.cpp.o:(.data.rel.ro._ZTV14TemplatedClassIiE[_ZTV14TemplatedClassIiE]+0x20): undefined reference to `TemplatedClass<int>::method()' collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. </code></pre> </details> Looking into the object for the new source file, you'll find the root cause: other_library.cpp.o symbols</a></summary> $ nm --demangle ./build/CMakeFiles/example.dir/other_library.cpp.o U operator delete(void, unsigned long) 0000000000000000 W ChildClass<int>::method() 0000000000000000 W ChildClass<int>::~ChildClass() 0000000000000000 W ChildClass<int>::~ChildClass() 0000000000000000 W ChildClass<int>::~ChildClass() 0000000000000000 n ChildClass<int>::~ChildClass() U TemplatedClass<int>::method() 0000000000000000 W TemplatedClass<int>::~TemplatedClass() 0000000000000000 W TemplatedClass<int>::~TemplatedClass() 0000000000000000 W TemplatedClass<int>::~TemplatedClass() 0000000000000000 n TemplatedClass<int>::~TemplatedClass() 0000000000000000 V typeinfo for ChildClass<int> 0000000000000000 V typeinfo for TemplatedClass<int> 0000000000000000 V typeinfo name for ChildClass<int> 0000000000000000 V typeinfo name for TemplatedClass<int> 0000000000000000 V vtable for ChildClass<int> 0000000000000000 V vtable for TemplatedClass<int> U vtable for cxxabiv1::class_type_info U vtable for cxxabiv1::si_class_type_info </code></pre> </details> There's an undefined reference to TemplatedClass<int>::method()</code>, which is what the linker is complaining about. It's a bit confusing that we have a hard dependency on a method that is never called, but after all, it is possible for a method override to call the parent's version of the method. But we've explicitly instantiated TemplateClass<int></code>, why hasn't method()</code> also been instantiated? Well, it's defined in library.cpp</code>, but our explicit instantiation is in other_library.cpp</code>. The definition is inaccessible! But how do we make the definition available without defining it in the header? Well, how about adding #include "library.cpp"</code> in other_library.cpp</code>? That's pretty much the only practical solution. I'm convinced I've seen this pattern once before, possibly in a standard library implementation. Typically, a different file extension is used to indicate it's an template implementation file, e.g. .tpp</code> or .txx</code>. Semantically, this feels better than #include</code>ing a .cpp</code> file in another. Let's apply this here, so let's rename library.hpp</code> to library.tpp</code>. We can also remove the explicit instantiation of the parent class, because that's implicit in the instantiation of the child class. See commit 8757c737 Resolve undefined parent class method (diff</a>, tree</a>) library.tpp</a></summary> 1</td> #include "library.hpp" </td></tr> 2</td> </td></tr> 3</td> template <typename T> </td></tr> 4</td> int TemplatedClass<T>::method() { return 42; } </td></tr></tbody></table></code></pre> </details> other_library.cpp</a></summary> 1</td> #include "other_library.hpp" </td></tr> 2</td> </td></tr> 3</td> #include "library.tpp" </td></tr> 4</td> </td></tr> 5</td> template <typename T> </td></tr> 6</td> int ChildClass<T>::method() { return 1; } </td></tr> 7</td> </td></tr> 8</td> template class ChildClass<int>; </td></tr></tbody></table></code></pre> </details> And our link issue is resolved: build log</a></summary> $ cmake -B build -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/avery/scratch/public/linking-by-example/build $ cmake --build build [1/3] Building CXX object CMakeFiles/example.dir/other_library.cpp.o [2/3] Building CXX object CMakeFiles/example.dir/main.cpp.o [3/3] Linking CXX executable example </code></pre> </details> Is this rare pattern worth introducing for a probably negligible impact on build times? Probably not. But should you encounter this situation in the future, now you know what to do! Hopefully you learned a thing or two along the way. What surprises me is that there's no compiler warning when an explicit instantiation can't generate all the code required for said instantiation. Personally I feel that's the intent behind an explicit instantiation, so the compiler should raise if it's unable to follow that instruction. It could also helpfully diagnose which definitions are missing. For the record, I've built these examples with GCC 13. Maybe things will improve in later versions. References</h2> ^1https://refspecs.linuxbase.org/elf/elf.pdf</a> </div> ^2https://en.cppreference.com/w/cpp/language/definition</a> </div> ^3https://en.cppreference.com/w/cpp/language/constexpr</a> </div> ^4https://en.cppreference.com/w/cpp/language/consteval</a> </div> ^5https://en.cppreference.com/w/cpp/language/templates</a> </div> Running NixOS on the Librem 5 Unknown — Fri, 03 Nov 2023 00:00:00 +0000 Introduction</h2> So, I got a used Purism Librem 5. Bought it used with the grand idea of actually doing something useful: contributing to the mobile GNU/Linux ecosystem. This was definitely inspired by Sophie Tauchert's talk</a> at NixCon 2023, where she goes over her work and the current status. I thought, since she's already upstreamed that work to nixos-hardware</a>, it should be easy enough to reproduce, right? NixOS seems a good match for a mobile phone. In particular, the ability to easily rollback to a previous known-good system configuration is a key feature. Traditionally this is done at the OS-level with an A/B partitioning system, but with NixOS this works for the entire system configuration. Theoretically, should I break things while I'm tinkering, it's easy to return to a known good state. Something I personally find important in my smartphone is stability. While I've dabbled plenty in custom ROMs, I've mostly stuck to well-supported and less flavourful distributions such LineageOS and its predecessor CyanogenMod. Ultimately I've never felt comfortable leaving my phone in an unusable state while tinkering with it, which has caused me plenty of stress while modding my phones. Less recently, I purchased another used Android smartphone to replace my LG G3 which had served me well for almost a decade at that point. With it, I could still meet my daily needs in a smartphone, but its age was starting to show. It had undergone a motherboard replacement, consumed a handful of (removable!) batteries, and still functions normally to this day. However it struggled with some of the more intensive applications - in particular Organic Maps</a>, it struggled to survive a full day's usage, and most importantly, custom ROM support had dried up, no longer being officially supported by LineageOS</a>. While I was tempted to dabble in the Android ROM development scene and bring it back to life, that seemed like a lot of work for a phone I'd most likely not use again. I can hardly hack on my daily driver, right? On top of that, I'm aware of what a mess Android distributions are. Kernel forks, non-mainlined drivers, proprietary firmware blobs. It takes a small army of volunteers to maintain all these devices - with a lot of redundant effort across ROM development groups - to keep these old devices running long after their manufacturers have dropped support. I'm immensely grateful for all these volunteers (especially HardStyl3r</a>, who supported the LG G3 across several versions of LineageOS), but I can't help viewing the Android ecosystem as a whole with a degree of fatalism. Google has made some attempts at improving the software support situation for Android devices, e.g. with Project Treble</a>, but in the years since, device manufacturers haven't improved their support guarantees much</a>. If, like myself, you choose to go through the rigmarole of granting yourself root access to fully control your device, you'll often find yourself locked out of automatic updates anyway. Being effectively punished for wanting to truly own my phone does not engender feelings of satisfaction. This is all while ignoring the hulking elephant in the room that is Google itself, whose motives and priorities are usually misaligned with my own. So, enter the extreme option: mobile GNU/Linux. I find this much more appealing, especially given I already use this on my desktops every day. Less proprietary, more Free, and independent from Google. Mobile GNU/Linux has picked up some momentum more recently, with manufacturers such as PINE64</a> and Purism</a> releasing smartphone with (mostly) open hardware with (almost) mainline Kernel support. Minimise the work needed for each new device - now that seems like a more sustainable approach. And so, as is commonly the case with me, I opted to act out of principle rather than practicality. My somewhat recent delve into Nix and NixOS</a> has given me the impression that it's a much better way to build software and GNU/Linux distributions. Theoretically, there's nothing I depend upon a smartphone for that couldn't run on GNU/Linux. Well, with the exception of my banking, but I'd happily drop my bank for one that allows me to access my account through a (relatively) more open platform like the Web. Starting point</h2> The phone arrived with Purism's PureOS distribution installed, version 10 (Byzantium). Full-disk encryption was enabled, with a pretty unlock screen on boot. Judging from the few megabytes of free space before the real partitions, u-boot was already installed (more on that later). It all seemed to be working normally, though I didn't have a spare activated SIM at hand to test telephony. A deactivated one I had lying around did get picked up, but of course couldn't connect to the network. It did briefly show 3G in the status bar, which sounded promising. The kill switch also appeared to work correctly: disabling and then re-enabling the switch made the mobile network grey out and then reconnect respectively. Unfortunately, when I remove the SIM tray the phone consistently rebooted, which I suspect is an intentional hardware feature. I ordered a new SIM from a cheap UK provider and left testing telephony for later. In Sophie's talk, she mentions the work she upstreamed into nixos-hardware, so that's where I started</a>. The first step mentions setting up Jumpdrive, a small system you can flash to the phone to expose its storage over USB. Given that the phone already had PureOS installed and functioning on it, I opted to skip this for now, as I could potentially install NixOS over SSH with nixos-anywhere</a>, discussed in another NixCon 2023 talk</a>. Fingers crossed I don't have to get into recovery mode at some point. The next step was to install u-boot - I suppose a different version than was already installed? nixos-hardware does not expose these packages as a regular Flake would, with the instructions using the traditional nix-build</code> instead of Flake-y nix build</code>. Not knowing how to enable cross compilation with nix-build</code>, I decided to make a quick and dirty Flake in my checkout of the repository: $ git clone https://github.com/NixOS/nixos-hardware $ cd purism/librem/5r4/ $ nix flake init $ cat flake.nix { description = "A very basic flake"; outputs = { self, nixpkgs }: { packages.aarch64-linux.u-boot = let pkgs = import nixpkgs { system = "aarch64-linux"; }; in pkgs.callPackage ./u-boot {}; }; } $ nix flake lock $ NIXPKGS_ALLOW_UNFREE=1 nix build --impure .#packages.aarch64-linux.u-boot </code></pre> I built it on my x86-64 desktop with aarch64 binfmt emulation enabled, it didn't take too long. The result is a script which writes the built image to the root of the main disk. To get this on the device, I simply copied the script and the referenced image over WiFi onto the device, then executed it: $ sudo TARGET=. ./u-boot-install-librem5 /dev/mmcblk0 </code></pre> There was some output from dd</code> which all indicated success. After a bit of fretful procrastination, I finally rebooted the device and... was pleasantly met with the full-disk encryption prompt as normal. Success! Building a configuration</h2> First things first: I needed a NixOS system configuration for the phone. I started off with the suggested approach of enabling Phosh, but quickly stumbled once I realised I had to deal with full-disk encryption, which I hadn't used before on NixOS. This is definitely a hard requirement for a smartphone personally - anyone grabbing my phone should not have immediate trivial access to it contents. The regular cryptsetup method suits me just fine, I won't worry about more in-depth systems like boot verification for now. On NixOS, cryptsetup is normally controlled with boot.initrd.luks.devices</a>, but I was unsure of how to actually input the decryption password. I was also unsure if simply plugging in a USB-C keyboard would work; my main keyboard does use USB-C, but it requires the hid_apple</code> kernel module to be enabled in the initial ramdisk. It does work in the existing PureOS installation, so it was probably excessive caution. In any case, having to plug in a keyboard to boot a phone is not ideal, so I continued researching entirely sure I dug around on PureOS and discovered that it's using Plymouth with an on-screen keyboard extension, osk-sdl</a>. An attempted package patch for nixpkgs was discarded</a>, with Samuel Dionne-Riel</a> mentioning the Mobile NixOS project had rejected osk-sdl</a>. Of course, I'd completely forgotten that the Mobile NixOS</a> project existed. Its main goals are providing a touchscreen-based boot process, and to automatically sort out any device quirks for you. The latter is already provided by Sophie's nixos-hardware module, but the former sounds like a great solution to me. Mobile NixOS doesn't currently have any support for the Librem 5, but I thought it shouldn't be too difficult to port it. My first evaluable system derivation is chronicled in 275174d3</a>. I don't think I quite got the hardware config right, but I wanted to get something building first. The configuration used nixpkgs unstable, as the Mobile NixOS docs stated</a>: Note: Mobile NixOS is only expected to build succesfully against the unstable branch of Nixpkgs </blockquote> $ nix build .#nixosConfigurations.maia.config.system.build.toplevel </code></pre> The first build took quite a while, with over 300 dependencies to be built. After getting through about 1000, it finally fell over on gdm</code>, the GNOME desktop manager. At this point my only solution seemed to be updating my nixpkgs pins. But knowing the Mobile NixOS project existed, and aarch64 was supported by NixOS, there was surely some nixpkgs commit where all these dependencies where already in the public Nix cache. And so to hydra I went, eventually finding the Mobile NixOS project, and found a recent build that evaluated the Phosh example successfulyl</a>. In the inputs tab</a> I found the nixpkgs commit used: 5e4c2ad</a>. So I updated my nixpkgs-unstable pin accordingly in 544a1ea6</a>: $ nix flake lock --override-input nixpkgs-unstable github:nixos/nixpkgs/5e4c2ada4fcd54b99d56d7bd62f384511a7e2593 --update-input nixpkgs-unstable </code></pre> I then hit another evaluation error: $ nix build .#nixosConfigurations.maia.config.system.build.toplevel error: systemdStage1 cannot be found in pkgs </code></pre> This confounded me for quite a while. Especially since I could find and build this in the pkgs</code> attribute of the derivation: nix-repl> :b outputs.nixosConfigurations.maia.pkgs.systemdStage1 :b outputs.nixosConfigurations.maia.pkgs.systemdStage1 </code></pre> The only useful information the trace (--show-trace</code>) gave me was that this was a dependency of plymouth</code>. After some unsuccessful attempts at debugging (the module system remains as opaque as ever) I eventually realised this was a problem with the NixOS system definition. It was being defined with the stable nixpkgs flake, but I was overriding pkgs</code> to use the imported unstable nixpkgs flake. I fixed the problem in 236b756c</a> by using the unstable nixpkgs lib</code> instead, leaving myself a note to research how nixpkgs works in NixOS configurations. Starting the build again, the dependencies were substituted from the cache instead of built locally! But then it hit the kernel, which I knew was going to be a little tough. Building the kernel</h3> The nixos-hardware repo contains a package for Purism's kernel fork for the Librem 5, so that's probably the kernel package I need to use. This is actually the first time I've tried to build Linux itself properly, aside from when I first tried to put NixOS on a Raspberry Pi. That first attempt, I ran into disk space - in reality, /tmp</code> space - due to the sheer quantity of code being compiled. In particular, I remember attempting to compile several GPU drivers - things that I definitely don't need on an ARM SOC. On top of that, Nix is compiling using an aarch64 emulator, which slows building down tremendously. Last time I hit the disk space issues I also recall researching and discovering that you can direct Nix where to put the build directory, which would be a workaround. It was something like TEMP</code>, TEMPFS</code>, TMPFS</code>. I try it out for a few builds but nothing seems to work - the build directories are generated in /tmp</code> regardless. So I first tried starting a build just to check it wouldn't immediately fail. It started building successfully, but slowly. At this point I figured I should turn on compilation caching for the derivation, in case I run into the same disk space issue. It's a bit confusing, but following the wiki</a> and a discourse thread</a>, I eventually got it working in ea9246cd</a>. Even with the compilation cache the build speed doesn't seem to be much improved, even when interrupting and re-running the start of the build. But hey, it's something, and I figure it may come in handy as a mess with the package. The next thing I want to figure out is how to turn off those drivers I definitely don't need. Time to give nix develop</code> a whirl: $ nix develop .#nixosConfigurations.maia.config.system.build.kernel $ mkdir /tmp/kernel $ unpackPhase unpacking source archive /nix/store/hsqvhjlchdc93mbmsaxjb7sm9vs3cks7-source source root is source $ cd ./source $ patchPhase applying patch /nix/store/23728y7zgh1jb55kpwxv5qnjbq0ykca6-randstruct-provide-seed-5.19.patch patching file scripts/gen-randstruct-seed.sh substituteStream(): WARNING: pattern '/bin/pwd' doesn't match anything in file 'Makefile' substituteStream(): WARNING: pattern '/bin/pwd' doesn't match anything in file 'tools/scripts/Makefile.include' patching script interpreter paths in scripts/ld-version.sh scripts/ld-version.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/4qp96hwq3wqkqhv99m76g2rp8sdjcl4v-bash-5.2-p15/bin/sh" patching script interpreter paths in scripts ... arch/arm64/boot/install.sh: interpreter directive changed from "#!/bin/sh" to "/nix/store/4qp96hwq3wqkqhv99m76g2rp8sdjcl4v-bash-5.2-p15/bin/sh" $ configurePhase no configure script, doing nothing $ buildPhase build flags: -j12 SHELL=/bin/bash O=\$$buildRoot$ CC=/nix/store/ahrxi3smaz2k98dsilzfjh5kbqya4bil-ccache-links-wrapper-4.8.3/bin/cc HOSTCC=/nix/store/ahrxi3smaz2k98dsilzfjh5kbqya4bil-ccache-links-wrapper-4.8.3/bin/cc HOSTLD=/nix/store/azdignpplcnd1gw87a3zb8nyabqnfqi9-binutils-wrapper-2.40/bin/ld ARCH=arm64 KBUILD_BUILD_VERSION=1-NixOS Image vmlinux modules dtbs DTC_FLAGS=-@ * * Configuration file ".config" not found! * * Please run some configurator (e.g. "make oldconfig" or * "make menuconfig" or "make xconfig"). * Makefile:770: include/config/auto.conf.cmd: No such file or directory make: * [Makefile:779: .config] Error 1 </code></pre> Okay, the build falls over immediately, meaning I've missed a step. The configuraPhase</code> doing nothing is suspicious, as normally the sequence of unpack, patch, configure, build should Just Work. The fact that configurePhase</code> bailed is suspicious. As I've never built the kernel manually before, I need to understand the build process before I can proceed. I start by investigating the regular derivation with nix edit nixpkgs#linux</code>. This opens up pkgs/os-specific/linux/kernel/mainline.nix</code>, which looks like some unhelpful boilerplate. But the directory looks about right, so I go up a level and inspect some of the neighbouring files. In generic.nix</code> I find more useful derivation derivation. It's more complex than most I've seen, which makes sense for the complexity of the kernel. There's a lot relating to configuration, which involves importing an auxiliary derivation in ./manual-config.nix</code>. It's strange that there's extra patches like this in the generic derivation, yet it doesn't seem to be applied in my source directory: { # ... postPatch = kernel.postPatch + '' # Patch kconfig to print "###" after every question so that # generate-config.pl from the generic builder can answer them. sed -e '/fflush(stdout);/i\printf("###");' -i scripts/kconfig/conf.c ''; } </code></pre> The Librem kernel derivation specifically uses the buildLinux</code> function, so I go hunting for it. nix edit nixpkgs#buildLinux</code> doesn't find anything. A simply search for buildLinux = </code> finds... { buildLinux = attrs: callPackage ../os-specific/linux/kernel/generic.nix attrs; } </code></pre> So I'm looking in the right place I guess. I look at the build log of my previous attempt: $ nix log /nix/store/7m0x5qgxw9g2xzbhvrn87izp0grnhpdx-linux-6.4.14-librem5.drv @nix { "action": "setPhase", "phase": "unpackPhase" } unpacking sources unpacking source archive /nix/store/hsqvhjlchdc93mbmsaxjb7sm9vs3cks7-source source root is source @nix { "action": "setPhase", "phase": "patchPhase" } patching sources ... @nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" } updateAutotoolsGnuConfigScriptsPhase @nix { "action": "setPhase", "phase": "configurePhase" } configuring ... </code></pre> There's a curious build phase I've never seen before. I run it in my source tree and get no output, same as the log. But configurePhase</code> still says no configure script, doing nothing</code>, what gives? configurePhase</code> has the log message at the end: $ type configurePhase configurePhase is a function configurePhase () { runHook preConfigure; : "${configureScript=}"; if [[ -z "$configureScript" && -x ./configure ]]; then configureScript=./configure; fi; if [ -z "${dontFixLibtool:-}" ]; then export lt_cv_deplibs_check_method="${lt_cv_deplibs_check_method-pass_all}"; local i; find -L . -iname "ltmain.sh" -print0 | while IFS='' read -r -d '' i; do echo "fixing libtool script $i"; fixLibtool "$i"; done; CONFIGURE_MTIME_REFERENCE=$(mktemp configure.mtime.reference.XXXXXX); find -L . -executable -type f -name configure -exec grep -l 'GNU Libtool is free software; you can redistribute it and/or modify' {} \; -exec touch -r {} "$CONFIGURE_MTIME_REFERENCE" \; -exec sed -i s_/usr/bin/file_file_g {} \; -exec touch -r "$CONFIGURE_MTIME_REFERENCE" {} \;; rm -I -f "$CONFIGURE_MTIME_REFERENCE"; fi; if [[ -z "${dontAddPrefix:-}" && -n "$prefix" ]]; then prependToVar configureFlags "${prefixKey:---prefix=}$prefix"; fi; if [[ -f "$configureScript" ]]; then if [ -z "${dontAddDisableDepTrack:-}" ]; then if grep -q dependency-tracking "$configureScript"; then prependToVar configureFlags --disable-dependency-tracking; fi; fi; if [ -z "${dontDisableStatic:-}" ]; then if grep -q enable-static "$configureScript"; then prependToVar configureFlags --disable-static; fi; fi; fi; if [ -n "$configureScript" ]; then local -a flagsArray; _accumFlagsArray configureFlags configureFlagsArray; echoCmd 'configure flags' "${flagsArray[@]}"; $configureScript "${flagsArray[@]}"; unset flagsArray; else echo "no configure script, doing nothing"; fi; runHook postConfigure } </code></pre> Which means that configureScript</code> must be undefined, and sure enough it is. But where should it come from, and what should it be defined to. For sanity I try a different approach, running nix develop .#nixosConfigurations.maia.pkgs.linuxPackages_librem5.kernel</code> instead. No dice, configurePhase</code> has the same output. Curiously this and .#nixosConfigurations.maia.config.system.build.kernel</code> are different derivations, I'm not sure why. At this point I give in and search up how others use Nix to hack on the kernel. I find this NixOS & Flakes Book</a> page with a sample flake. The key difference with their approach is in their use of linuxPackages_thead.kernel.dev</code>, the dev</code> attribute of the derivation. I try nix develop ~/.config/nix#nixosConfigurations.maia.config.system.build.kernel.dev</code> but there's no difference in running configurePhase</code>. Another thing that's confusing me is the different between the configurePhase</code> function and $configurePhase</code> variable: $ echo $configurePhase runHook preConfigure mkdir build export buildRoot="$(pwd)/build" echo "manual-config configurePhase buildRoot=$buildRoot pwd=$PWD" if [ -f "$buildRoot/.config" ]; then echo "Could not link $buildRoot/.config : file exists" exit 1 fi ln -sv /nix/store/4hvv1ywdp5k7p0mb4sd9azhywgqpb8cv-linux-config-6.4.14-librem5 $buildRoot/.config # reads the existing .config file and prompts the user for options in # the current kernel source that are not found in the file. make $makeFlags "${makeFlagsArray[@]}" oldconfig runHook postConfigure make $makeFlags "${makeFlagsArray[@]}" prepare actualModDirVersion="$(cat $buildRoot/include/config/kernel.release)" if [ "$actualModDirVersion" != "6.4.14-librem5" ]; then echo "Error: modDirVersion 6.4.14-librem5 specified in the Nix expression is wrong, it should be: $actualModDirVersion" exit 1 fi buildFlagsArray+=("KBUILD_BUILD_TIMESTAMP=$(date -u -d @$SOURCE_DATE_EPOCH)") cd $buildRoot </code></pre> The variable contains that manual-config</code> line that is in the derivation, but running it produces no output if I run $configurePhase</code>. (set -x; $configurePhase)</code> shows a bunch of flags being built before an exit 0</code>. This is driving me a bit mad. Let's try building the kernel the traditional way. Building the kernel, take 2</h3> Some searching brings me to the Kernel docs</a> which link to this "admin guide"</a>. The file path and title are a bit misleading, but the breadcrumb says The Linux kernel user’s and administrator’s guide, which is a bit more accurate. This guide soon directs you to build menuconfig</code>, a tool for configuring the kernel. It quickly falls over due to ncurses</code> being unavailable. $ make O=build/kernel menuconfig GEN Makefile HOSTCC scripts/basic/fixdep HOSTCC scripts/kconfig/mconf.o In file included from /tmp/kernel/source/scripts/kconfig/mconf.c:23: /tmp/kernel/source/scripts/kconfig/lxdialog/dialog.h:19:10: fatal error: ncurses.h: No such file or directory 19 | #include <ncurses.h> | ^~~~~~~~~~~ compilation terminated. make[2]: * [/tmp/kernel/source/scripts/Makefile.host:131: scripts/kconfig/mconf.o] Error 1 make[1]: * [/tmp/kernel/source/Makefile:692: menuconfig] Error 2 make: * [Makefile:226: __sub-make] Error 2 </code></pre> Which means I need to have ncurses</code> as a shell input. It's rather difficult to make an impromptu shell including a particular package for development, so I think I have to bodge one into my Flake. I end up overriding my existing override for the kernel package to add the build input: { # ... cached_linuxPackages_librem5 = (super.linuxPackages_librem5.kernel.override { stdenv = self.ccacheStdenv; buildPackages = super.buildPackages // { stdenv = self.ccacheStdenv; }; }).overrideAttrs(old: { buildInputs = [super.ncurses]; }); } </code></pre> Nastier, but easier. I nix develop</code> into the configuration attribute and now the build proceeds further, but fails with missing symbol errors: $ make O=build/kernel menuconfig GEN Makefile HOSTCC scripts/kconfig/mconf.o HOSTCC scripts/kconfig/lxdialog/checklist.o HOSTCC scripts/kconfig/lxdialog/inputbox.o HOSTCC scripts/kconfig/lxdialog/menubox.o HOSTCC scripts/kconfig/lxdialog/textbox.o HOSTCC scripts/kconfig/lxdialog/util.o HOSTCC scripts/kconfig/lxdialog/yesno.o HOSTCC scripts/kconfig/confdata.o HOSTCC scripts/kconfig/expr.o LEX scripts/kconfig/lexer.lex.c YACC scripts/kconfig/parser.tab.[ch] HOSTCC scripts/kconfig/lexer.lex.o HOSTCC scripts/kconfig/menu.o HOSTCC scripts/kconfig/parser.tab.o HOSTCC scripts/kconfig/preprocess.o HOSTCC scripts/kconfig/symbol.o HOSTCC scripts/kconfig/util.o HOSTLD scripts/kconfig/mconf /nix/store/as2izmyw2771n7p60nlm0f6a84kmnyin-binutils-2.40/bin/ld: scripts/kconfig/mconf.o: in function `show_help': mconf.c:(.text+0xa18): undefined reference to `stdscr' /nix/store/as2izmyw2771n7p60nlm0f6a84kmnyin-binutils-2.40/bin/ld: mconf.c:(.text+0xa20): undefined reference to `stdscr' ... </code></pre> I took a brief diversion to test whether this is an issue with cross-compiling by testing out building the kernel on the native x86-64 architecture. I added the kernel to my Flake's packages like so: { # ... linuxPackages_librem5 = pkgs.cached_linuxPackages_librem5; } </code></pre> I can't even open the dev shell however, because the config dependency fails to build: $ nix develop ~/.config/nix#linuxPackages_librem5 warning: Git tree '/home/avery/.config' is dirty error: builder for '/nix/store/m2wx8d43gnqhjg9k4khwpdvy97224gfj-linux-config-6.4.14-librem5.drv' failed with exit code 2; last 10 log lines: > HOSTCC scripts/kconfig/symbol.o > HOSTCC scripts/kconfig/util.o > HOSTLD scripts/kconfig/conf > * > * Can't find default configuration "arch/x86/configs/librem5_defconfig"! > * > make[2]: * [../scripts/kconfig/Makefile:94: librem5_defconfig] Error 1 > make[1]: * [/build/source/Makefile:692: librem5_defconfig] Error 2 > make: * [Makefile:226: __sub-make] Error 2 > make: Leaving directory '/build/source' For full logs, run 'nix log /nix/store/m2wx8d43gnqhjg9k4khwpdvy97224gfj-linux-config-6.4.14-librem5.drv'. error: 1 dependencies of derivation '/nix/store/s05nyymk5bsf7gmxfzl1mbas4y7m23lr-linux-6.4.14-librem5-env.drv' failed to build </code></pre> Clearly the configuration file is only specified for the aarch64 architecture, and sure enough I find it under arch/arm64/configs/librem5_defconfig</code>. This does complicate things, if I want to build this package I'll have to patch the package. I tried to extend postPatch</code> in my overlay: { # ... cached_linuxPackages_librem5 = (super.linuxPackages_librem5.kernel.override { stdenv = self.ccacheStdenv; buildPackages = super.buildPackages // { stdenv = self.ccacheStdenv; }; }).overrideAttrs(old: { buildInputs = [super.ncurses]; postPatch = old.postPatch + '' cp arch/arm64/configs/librem5_defconfig arch/x86/configs/ ''; }); } </code></pre> But this failed the same as before. Apparently this isn't enough to override the config derivation. I do a bit more reading and digging around, and find out that these config values are "tristate", being either y</code> (yes, on), no</code> (no, off), or m</code> (module, as in loaded separately at runtime as a kernel module). I find the appropriate config in drivers/gpu/drm/amd/amdgpu/Kconfig</code>: DRM_AMDGPU</code>. So I want to set CONFIG_DRM_AMDGPU=n</code> somehow. At this point I noticed something in the generic Linux derivation: { # ... # Adds dependencies needed to edit the config: # nix-shell '<nixpkgs>' -A linux.configEnv --command 'make nconfig' configEnv = kernel.overrideAttrs (old: { nativeBuildInputs = old.nativeBuildInputs or [] ++ (with buildPackages; [ pkg-config ncurses ]); }); } </code></pre> permalink</a> This... sounds like exactly what I need. I couldn't find it in the system.build.kernel</code> attribute, but it is in the package proper: $ nix develop ~/.config/nix#nixosConfigurations.maia.config.system.build.kernel.configEnv $ make V=1 O=build/kernel nconfig </code></pre> Huzzah! I get a beautiful ncurses GUI. In this GUI I can load the Librem 5 config file, and navigating to Device Drivers > Graphics support</code> I do indeed see the AMDGPU and Radeon drivers enabled. I try saving my configuration to compare it to the existing one, but wherever nconfig is saving them, it isn't in the working directory. Turns out it was in the build directory, because I was running it with make</code>. The new config is very different to the existing one, being more verbose and thus less diffable. But I do find this in it: # CONFIG_DRM_AMDGPU is not set </code></pre> I guess it's off by default? I don't really get why it's being built then. On another note, I find the NixOS package has the argument structuredExtraConfig</code>, which lets you specify config overrides. Handy. I think I'll give that a shot and give up on building the kernel manually. { # ... cached_linuxPackages_librem5 = super.linuxPackages_librem5.kernel.override { stdenv = self.ccacheStdenv; buildPackages = super.buildPackages // { stdenv = self.ccacheStdenv; }; structuredExtraConfig = { CONFIG_DRM_AMDGPU = "n"; CONFIG_DRM_RADEON = "n"; CONFIG_DRM_NOUVEAU = "n"; }; }; } </code></pre> Okay, let's give building another shot... once I figure out how to change the build directory. Changing the build directory</h3> Going off my previous knowledge, I search for TMPDIR</code> in the NixOS Matrix rooms and find this message</a> in the NixOS ARM Matrix room. Turns out I was right, that was the environment variable I needed, but the key thing I missed was that this doesn't affect builds run through the Nix daemon. This includes building as my unprivileged user with Nix installed in multi-user mode. A workaround therefore, is to run the build as root. Not ideal, but it'll do for now: $ sudo TMPDIR=/data/random/nix nix build ~/.config/nix#nixosConfigurations.maia.config.system.build.kernel $ ls /data/random/nix nix-build-linux-6.4.14-librem5.drv-0 </code></pre> Finally! Now, time to wait for the build to complete. Halfway through I dig into the build directory and find the config file under /source/build/.config</code>. That the AMDGPU etc. options are still set to m</code>, hmm. I inspect the Librem buildLinux</code> call and see that they don't have the CONFIG_</code> prefix. Whoops. Time to try again. Hopefully that build cache works. { # ... cached_linuxPackages_librem5 = super.linuxPackages_librem5.kernel.override { stdenv = self.ccacheStdenv; buildPackages = super.buildPackages // { stdenv = self.ccacheStdenv; }; structuredExtraConfig = { DRM_AMDGPU = "n"; DRM_RADEON = "n"; DRM_NOUVEAU = "n"; }; }; } </code></pre> I try again but the config file is wrong, again. $ nix eval ~/.config/nix#nixosConfigurations.maia.config.system.build.kernel.structuredExtraConfig warning: Git tree '/home/avery/.config' is dirty { CMA_SIZE_MBYTES = { _type = "override"; content = { freeform = "320"; optional = false; }; priority = 50; }; } </code></pre> This is clearly wrong - somehow my override isn't being applied. I faff around a bunch with builtins.trace</code>, during which I find stdenv</code> is being overridden correctly. I eventually notice some weirdness around the Librem kernel derivation involving overrides, and particularly this comment: structuredExtraConfig = with lib.kernel; { # buildLinux overrides this and defaults to 32, so go back to the value defined librem5_defconfig # this is required for millipixels to take photos, otherwise the VIDIOC_REQ_BUFS ioctl returns ENOMEM CMA_SIZE_MBYTES = lib.mkForce (freeform "320"); }; </code></pre> So they had to perform some hijinks to force an option value. Thankfully they left in a workaround: { lib , buildLinux , fetchFromGitLab , ... } @ args: buildLinux (args // rec { defconfig = "librem5_defconfig"; version = "6.4.14-librem5"; modDirVersion = version; src = fetchFromGitLab { domain = "source.puri.sm"; owner = "Librem5"; repo = "linux"; rev = "pureos/6.4.14pureos1"; hash = "sha256-PzRG6czWLMahklceuaWGK1QJ+m9FAKDa/m1jp87h62k="; }; kernelPatches = [ ]; structuredExtraConfig = with lib.kernel; { # buildLinux overrides this and defaults to 32, so go back to the value defined librem5_defconfig # this is required for millipixels to take photos, otherwise the VIDIOC_REQ_BUFS ioctl returns ENOMEM CMA_SIZE_MBYTES = lib.mkForce (freeform "320"); }; } // args.argsOverride or { }) </code></pre> I try using argsOverride</code> instead: { # ... cached_linuxPackages_librem5 = (super.linuxPackages_librem5.kernel.override (old: { stdenv = self.ccacheStdenv; buildPackages = super.buildPackages // { stdenv = self.ccacheStdenv; }; argsOverride.structuredExtraConfig = builtins.trace "structuredExtraConfig" (with super.lib.kernel; { DRM_AMDGPU = no; DRM_RADEON = no; DRM_NOUVEAU = no; }); })); } </code></pre> And it works! The configuration rebuilds itself, then the full kernel build begins proper. It's noticeably slower than before. Inspecting the compiler cache, I can see a tiny number of hits: $ nix run nixpkgs#ccache -- -s -d /data/random/ccache/ Cacheable calls: 1037 / 1766 (58.72%) Hits: 10 / 1037 ( 0.96%) Direct: 10 / 10 (100.0%) Preprocessed: 0 / 10 ( 0.00%) Misses: 1027 / 1037 (99.04%) Uncacheable calls: 729 / 1766 (41.28%) Local storage: Cache size (GiB): 3.7 / 5.0 (74.64%) Hits: 10 / 1037 ( 0.96%) Misses: 1027 / 1037 (99.04%) </code></pre> A shame, but fingers crossed the build finishes this time. The first attempt failed because my root disk filled up while the result was being copied to the Nix store. After a nix store gc</code> and cleanup, it succeeds! Thankfully the compilation cache sped things up quite a bit on the second attempt. Now that I have a built system configuration, time to figure out how to install it. Installing</h2> The nixos-hardware README</a> suggests flashing Jumpdrive to mount the internal storage on another device, but I'd prefer to avoid flashing if possible. Trying out nixos-anywhere and disko</h3> So I go back and look into nixos-anywhere</a> a bit more deeply. The quickstart guide</a> looks pretty simple: build a NixOS configuration, then give it an SSH connection and it does all the rest. Of course, it wasn't going to be that simple. I already had a building configuration, so I tried the nixos-anywhere</code> command immediately. This first attempt was met with error: flake 'git+file:///home/avery/.config?dir=nix' does not provide attribute 'packages.x86_64-linux.nixosConfigurations."maia".config.system.build.diskoScript', 'legacyPackages.x86_64-linux.nixosConfigurations."maia".config.system.build.diskoScript' or 'nixosConfigurations."maia".config.system.build.diskoScript' </code></pre> So clearly I'd missed a step. And indeed an earlier step mentions including the disko</a> module for automatic disk partitioning. Problem is, the phone's disk was already reasonably partitioned, so I'd really prefer if I could skip the partitioning step. After a bit of digging around I failed to find this as an option, so I gave it and gave disko a shot. I do quite like the idea of declarative partitioning, but it's rather disconcerting off the bat - when exactly does the partitioning take place? What if I add more partitions in the future - will it reformat everything automatically? A bit of reading around suggests it's only done with the nixos-anywhere</code> command or manually with some generated scripts</a>. Okay, fine, let's convert the existing configuration. purism@pureos:~$ sudo fdisk -l Disk /dev/mmcblk0: 29.12 GiB, 31268536320 bytes, 61071360 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x2c38c7d5 Device Boot Start End Sectors Size Id Type /dev/mmcblk0p1 * 10240 962559 952320 465M 83 Linux /dev/mmcblk0p2 962560 61071326 60108767 28.7G 83 Linux Disk /dev/mmcblk0boot0: 4 MiB, 4194304 bytes, 8192 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mmcblk0boot1: 4 MiB, 4194304 bytes, 8192 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mtdblock0: 192 KiB, 196608 bytes, 384 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mtdblock1: 1.81 MiB, 1900544 bytes, 3712 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/crypt_root: 28.65 GiB, 30758911488 bytes, 60075999 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/zram0: 1.45 GiB, 1555038208 bytes, 379648 sectors Units: sectors of 1 * 4096 = 4096 bytes Sector size (logical/physical): 4096 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes </code></pre> What I ended up with is in b5707ab6</a>. As expected, figuring out the edge cases was a bit weird. Not knowing too much about disk partitioning made things harder to work out. For the LUKS setup, there were plenty of examples to look at, but I couldn't quite figure out how to set the cipher for the disk declaratively. Turns out the extraFormatArgs</code> option lets you do that. I really missed having something like the NixOS options search</a> to explore the option. Tab completion in the nix repl</code> really doesn't cut it, especially with heavy usage of submodule option types in the disko module. The lack of option documented is something the disko project is aware of</a>, but the examples don't quite cover this case. Mental note: explore a Nix language server as mentioned in this NixCon 2023 talk</a>. The other thing I found difficult to work out was the DOS mode (aka. MBR/BIOS mode?), as the fdisk</code> output includes Disklabel type: dos</code>. I do recall this when installing Arch Linux manually - GPT is the more modern thing relating to the UEFI boot mode. Searching through the disko source, I sometimes saw msdos</code> besides gpt</code>. So I thought I simply had to change the type: "gpt"</code> in the main disk attrset to msdos</code>. That failed to build, so I eventually found this "compat" example</a> for supporting both boot modes. This looks similar to the existing partition scheme, with a traditional boot EFI partition. The boot partition notably leaves some free space before it, 10240</code> bytes, for the MBR and u-boot I expect. In the disko example, it leaves just one MB. The Librem README suggests 2MB, but I double it to 4MB to be on the safe side. I also found this "standalone" example</a> which uses BIOS mode only, but this is quite different from the existing scheme, so I ignored it. It did lead me to research the compat mode however. I then built the diskoScript</code> attribute directly to test that out, but ran into a ShellCheck build error. Good to see they're checking their scripts, but apparently I've run into something which the project doesn't test for. I cloned the project locally, fixed it, and locked my flake to use the local checkout instead as a quick workaround. I later raised a quick patch</a>, which was very promptly merged. The nixos-anywhere guide mentions the --build-vm</code> option for testing out the configuration. I did try running this with nixos-anywhere --flake .#maia --vm-test</code>. Unfortunately it to a very long time, and I eventually aborted it. It seems to involve booting NixOS several times, I assume once with an initial NixOS image to partition the disk, then again once everything's installed. The configuration being an emulated aarch64</code> definitely contributed to it being so slow. I'm still not sure if I got the partitioning right, since I couldn't build the VM. Failed attempt: ... + /root/kexec/kexec/kexec --load /root/kexec/kexec/bzImage --kexec-syscall-auto --initrd=/root/kexec/kexec/initrd --no-checks --command-line init=/nix/store/bn1085m7ssvj31a065krrhn827sxz2fq-nixos-system-nixos-23.05pre-git/init console=tty0 console=ttyAMA0,115200 console=ttyS0,115200 loglevel=4 Can't open (/proc/kcore). Warning, can't get the VA_BITS from kcore Can't open (/proc/kcore). kexec_load failed: Function not implemented entry = 0x575e46a0 flags = 0xb70000 nr_segments = 4 segment[0].buf = 0xffff7aa17100 segment[0].bufsz = 0x379d200 segment[0].mem = 0x41000000 segment[0].memsz = 0x3860000 segment[1].buf = 0xffff67c83170 segment[1].bufsz = 0x12d7571c segment[1].mem = 0x44860000 segment[1].memsz = 0x12d76000 segment[2].buf = 0xffff67c67190 segment[2].bufsz = 0xda37 segment[2].mem = 0x575d6000 segment[2].memsz = 0xe000 segment[3].buf = 0xffff67c621b0 segment[3].bufsz = 0x3558 segment[3].mem = 0x575e4000 segment[3].memsz = 0x4000 + echo kexec failed, dumping dmesg ... </code></pre> After the fact I find an (as of writing) open PR</a> to explicitly skip the disko partitioning, which I think is a step in the right direction. With Jumpdrive</h3> Initially tried building it myself following the repository's instructions. But I quickly ran into issues when compiling the arm-trusted-provider dependency of u-boot. These (as far as I can tell) incorrect errors about dereferencing likely-zero pointers, but these were hard-coded addresses which definitely weren't 0. I overrode build flags in the Makefile, setting CFLAGS=-Wno-error</code>. But it failed anyway, with an error I couldn't understand. So I tried replacing it with the u-boot from the package in the nixos-hardware repo, moving things around to more or less reproduce the layout the uuu</code> script expected. I had to try a different USB-C cable before uuu</code> detected the device. And also sudo, I think. This seemed to work until it got to an unzip</code> command to extract the kernel. After several attempts, I give up and consider my bodge a failure, possibly due to the version of u-boot I flashed. Then I realised Jumpdrive already provides pre-built archives for the Librem 5</a>, so I just used that. And it actually worked this time. Thankfully Jumpdrive doesn't overwrite the internal partitions, it simply uploads and boots a tiny GNU/Linux operating system, exposing the internal storage over USB without touching it. Cool, the internal storage of the device now shows up as /dev/sdc</code> or /dev/sdd</code> as expected. I run the regular nixos-install</code> command to install the built configuration: $ nix build .#nixosConfigurations.maia.config.system.build.toplevel --out-link configurations/maia/result $ sudo cryptsetup open /dev/sdc2 phone_root ... $ sudo mount /dev/mapper/phone_root /mnt $ sudo mount /dev/sdc1 /mnt/boot $ sudo nixos-install -v --system ./configurations/maia/result --root /mnt ... Warning: do not know how to make this configuration bootable; please enable a boot loader. /nix/var/nix/profiles/system/sw/bin/bash: line 10: umount: command not found </code></pre> Guess I messed up the bootloader config. It was quite confusing to figure out what I should put into /boot</code>. It's not done automatically by nixos-install</code>. I tried enabling GRUB with boot.loader.grub.enable = true</code>, but that rendered the device unbootable. There's a lot of mention of the special stage-1 for Mobile NixOS, but little mention of how to actually install it. E.g. going off the Pinephone README</a>, the options mentioned are 1) flashing the installer image, and 2) flashing a full disk image. There's a mention of flashing only the boot partition, but the link is dead. I find this issue</a> which mentions a boot-partition</code> attribute. That attribute no longer exists, but I find similar ones by poking into the system configuration: $ nix build .#nixosConfigurations.maia.config.mobile.outputs.u-boot.boot-partition warning: Git tree '/home/avery/.config' is dirty error: cannot coerce null to a string at /nix/store/fkf6sxbwp0phwbrlw7n8bj8s5lpl6y3k-source/modules/system-types/u-boot/default.nix:13:18: 12| kernel = stage-0.mobile.boot.stage-1.kernel.package; 13| kernel_file = "${kernel}/${if kernel ? file then kernel.file else pkgs.stdenv.hostPlatform.linux-kernel.target}"; | ^ 14| inherit (config.mobile.generatedFilesystems) rootfs; (use '--show-trace' to show detailed </code></pre> Promising, but the kernel is missing. This must be set especially for the device's mobile config. I find it in the Pinephone config, so I must have to do this manually too: mobile.boot.stage-1.kernel.package = pkgs.cached_linuxPackages_librem5; </code></pre> And now the image builds. An ext4 image, fine. I write it with the suggested command: sudo dd if=result of=/dev/<disk>1 bs=8M oflag=sync,direct status=progress </code></pre> Moment of truth - I reboot, and... it works! I see a boot log! And eventually, a decryption prompt! Sadly the touchscreen keyboard doesn't work. Or a keyboard for that matter, not sure why that's the case. The prompt acts as if I pressed enter when I press the power button, funny. Digging into the boot log through the low-tech method of recording it with another phone, I see lots of errors. Right at the start is a bit suspect: Running Tasks::Environment... /proc/mounts: _get_sysfs_dir fopen failed: No such file or directory </code></pre> That's followed by a bunch of modprobe errors, including core modules like devpts</code>, devtmpfs</code>, proc</code>, tmpfs</code>, sysfs</code>, dm_mod</code>. All of the errors say not found in directory /lib/modules/6.4.14-librem5</code>. Quite strange, sounds like no modules are available? I tried comparing the kernel configuration in Mobile NixOS</a> to the config for my kernel. I noticed some possibly related config options, so added them to my kernel's config overrides: cached_linuxPackages_librem5 = (super.linuxPackages_librem5.kernel.override (old: { stdenv = self.ccacheStdenv; buildPackages = super.buildPackages // { stdenv = self.ccacheStdenv; }; argsOverride.structuredExtraConfig = with super.lib.kernel; { DRM_AMDGPU = no; DRM_RADEON = no; DRM_NOUVEAU = no; KERNFS = yes; SYSFS = yes; TMPFS = yes; }; })); </code></pre> And after a long kernel recompilation, I flashed the boot image again, and... no change. Clearly something else was going wrong. So I assume kernel modules live in the initial ramdisk, so I tried building that separately and extracting it. After a bunch of reading of the mobile-nixos source, I eventually found the attribute for it: $ nix build .#nixosConfigurations.maia.config.system.build.initialRamdisk $ file result/initrd result/initrd: gzip compressed data, max compression, from Unix, original size modulo 2^32 43448320 </code></pre> I extracted this file, which produced this a CPIO archive, and doing some quick searching found out how to extract it</a>: $ file /tmp/initrd /tmp/initrd: ASCII cpio archive (SVR4 with no CRC) $ cpio -idm -F /tmp/initrd 84850 blocks $ tree -L 2 . ├── applets │ ├── boot-error.mrb -> /nix/store/w6jryprm0r9q64c9jb91zbzj94dkpc2h-boot-error.mrb/libexec/boot-error.mrb │ ├── boot-selection.mrb -> /nix/store/4rsrp0flb19jvm3wzm8m50xmvkl6khbf-boot-recovery-menu.mrb/libexec/boot-recovery-menu.mrb │ └── boot-splash.mrb -> /nix/store/wni188k1gghr8rf4c1r6lp9gpayzd2mw-boot-splash.mrb/libexec/boot-splash.mrb ├── bin │ └── sh -> /nix/store/6rpafppbl0y5i7j0j3xszy173iqx7isg-extra-utils-purism-librem5-extra-utils/bin/sh ├── dev ├── etc │ ├── boot │ ├── logo.svg -> /nix/store/y9nh2w9bqnl2h9fwdv5wci8hqpc2wj70-logo.white.svg │ ├── udev │ └── X11 -> /nix/store/cng1lrqy9d11x2w81fdvv4dpkwvf76cb-minimalX11Config ├── init -> /nix/store/sxf72hnc9kqrrw5ndz12cqz4cqakz3h0-init-wrapper ├── init.mrb -> /nix/store/azq5psyan11i8ly2n9r5racdh6hs5m5n-mobile-nixos-init-0.1.0/libexec/init.mrb ├── lib │ ├── firmware -> /nix/store/fkvh8bmwyccylc540vvyk2h00kfjphbd-firmware/lib/firmware │ └── modules -> /nix/store/bjkk524f7d3yphibd9kdd8z0w04acirc-null-modules/lib/modules ├── loader -> /nix/store/6rpafppbl0y5i7j0j3xszy173iqx7isg-extra-utils-purism-librem5-extra-utils/bin/loader ├── nix │ └── store ├── proc └── sys </code></pre> So, lib/modules</code> sounds like the cause of the problem. It points to ...-null-modules/</code>, which both sounds wrong and is empty. Some more digging in the mobile-nixos sources, I found the null-modules</code> thing is related to the mobile.boot.stage-1.kernel.modular</code> option, which for some reason is false</code> by default. If I set it to true</code>, the build log suggests some modules are actually being built. I also try mobile.boot.stage-1.useNixOSKernel = true;</code>, since I am, after all using a kernel from the regular NixOS configuration. Unfortunately, the same errors occur, and the touchscreen still isn't usable. However... plugging in a USB-C keyboard actually lights up the keyboard and notification light. I think the USB drivers are working now, but I still can't type. I try a phone dock, but it's the same story. But I give it one last shot, typing with the dock as the phone boots, and it works! I enter the encryption key and it proceeds to NixOS stage 2. Things look good until it gets to the systemd "Show Plymouth Boot Screen" service, which seems to get stuck . Then it leaves me with the following: Cannot open access to console, the root account is locked. See sulogin(8) man page for more details. </code></pre> Still, this is promising. I'd like to at least get the phone booting, even if the touchscreen unlock isn't currently working. During the boot process, I see an error when systemd tried to mount /boot</code>: couldn't mount because of unspported optional features (40) </code></pre> Turns out it's because the filesystem from the generated boot partition is ext4, not ext2. Whoops. Amended, re-ran nixos-install</code>, and finally... it boots to the UI! It looks very similar to how it looked on PureOS. I swipe up to the unlock prompt... and realise I never set the password. One final reflash to Jumpdrive, nixos-enter</code>, passwd</code>, and a reboot, and finally I can unlock the device into the Phosh desktop. Fixing the touchscreen in stage 1</h3> The touchscreen not working for the unlock prompt was a rather large usability issue - I didn't want to keep plugging in a USB keyboard and hoping that the appropriate keyboard driver would actually be loaded in time. I assumed this was a case of the necessary kernel module not being loaded in the initramfs. In some initial research, I came across the Multitouch displays</a> page on the ArchWiki. This one mentions the hid_multitouch</code> module, but this one isn't loaded on the booted device, according to lsmod</code>. The Touchscreen</a> page mentions cat</code>ing the /dev/inputs/event</code> files while touching the screen until you find the one for the touchscreen. Unfortunately I forgot how these special files work and used tail -f</code> on them, and confused myself by finding that none of them produced output. Thankfully I had more luck with the other method, reading /proc/bus/input/devices</code>. None of the entries have "touch" in the name, but this one stuck out with the obscure name and the fact is has a mouse output: I: Bus=0018 Vendor=0000 Product=0000 Version=0000 N: Name="EP0700M09" P: Phys= S: Sysfs=/devices/platform/soc@0/30800000.bus/30a40000.i2c/i2c-2/2-0038/input/input5 U: Uniq= H: Handlers=event5 mouse1 B: PROP=2 B: EV=b B: KEY=400 0 0 0 0 0 B: ABS=260800000000003 </code></pre> A quick search for EP0700M09 brought up a schematic</a> from the "Emerging Display Technologies Corp", so this must be the display. And indeed cat /dev/input/event5</code> did show events while touching the screen. And finally, after searching "linux find module for device", this answer</a> led to: $ udevadm info -a -n /dev/input/event5|ag DRIVERS DRIVERS=="" DRIVERS=="edt_ft5x06" DRIVERS=="" DRIVERS=="imx-i2c" DRIVERS=="" DRIVERS=="" DRIVERS=="" </code></pre> edt</code> = Emerging Display Technologies? That sounds about right. I added edt_ft5x06</code> to boot.initrd.kernelModules</code> and rebooted, and found I could finally type on the on-screen keyboard! First impressions of Mobile NixOS</h2> I didn't have to go through the fancy setup that PureOS had - unlocking immediately showed the Phosh UI. Said UI looks very similar to the PureOS version, I guess not much has changed in recent updates. There's notably fewer apps preinstalled, really the bare minimum. A dialer, texts, browser (Epiphany), terminal, and camera app (Megapixels). I also installed a few more applications from the GNOME core suite with my home manager configuration</a>. I'll intersperse this section with some pretty screenshots. Mobile features</h3> Obviously one of the primary functions of a phone is texting and calling, so I wanted to test those out. At this point I'd received a SIM from one of the "cheaper" pay-as-you-go providers in the UK. Annoyingly the phone once again rebooted when I inserted the SIM card - this may be a hardware "feature" in the end. However after rebooting, I was quite surprised to almost immediately receive a text from the provider with registration instructions - that meant the modem automatically registered itself as you'd expect. Interestingly the settings defaulted to 2G & 3G only, and I had to manually enable 2G, 3G, and 4G. Reception seemed reasonable for being indoors. Mobile data took a bit more work, requiring me to manually set up the Access Point Names, by searching up my provider's details on another computer. After setting up the APNs, the 4G icon popped up and I could load websites at what seemed like reasonable speeds. I didn't spend too long testing it because of the overpriced mobile data usage costs. I then tested sending a text to my primary phone, which worked perfectly fine. Phone calls were less functional, I couldn't hear my voice on the Librem nor my Android phone, no matter what audio devices I selected on the Librem. Interestingly if I selected the Analog Output - Modem device, opened the regular GNOME sound test menu, and clicked the front left speaker, I heard it on the other side of the call! That seems a bit backwards, but whatever. I didn't want to spend too long on the initial test. Quick aside/rand about UK providers: that single text and a sub 3 minute call cost me a whole 78 pence! That's 10p for the text, and 25p per minute. I'd decided to put just £10 on a pay-as-you-go SIM because I didn't expect to be using it enough to justify a £10/mo contract, but I'm wondering just how long that'll last now. Phone carriers continue to be complete rip-offs, I see. I'll probably look for a better value PAYG SIM after I've exhausted this credit. Phosh</h3> Phosh's UI borrows heavily from Android et al., which is perfectly reasonable, and it looks pretty slick by itself. It feels smooth to swipe around, which is naturally an important feature for a touchscreen interface. I think it looks pretty slick too, and it even makes use of your GTK theme, allowing for some customisability. </a> A reasonably contemporary design for a smartphone lockscreen, if minimal.</figcaption> </figure> </a> Similar story for the unlock screen.</figcaption> </figure> One thing really irks me with Phosh's unlock screen, through no fault of its own. Android has conditioned me to expect the "enter" button in the bottom right of the keypad. However, in Phosh this is where the backspace button lives, and the unlock button instead lives at the bottom of the screen. I would love if this were configurable for poor Android users like myself. Another issue I've noticed is that you can swipe down on the keypad to return to the lock screen, which I've accidentally triggered multiple times while typing in my passcode. A simple solution would be adding an element behind the keypad which consumes touch inputs, and have the keypad buttons themselves consume the swipe inputs. </a> The main Phosh UI after unlocking the screen. Reminds me of a combination of the Android recent apps screen and app drawer.</figcaption> </figure> An interesting choice was to combine the recent apps with the app drawer. I don't particularly disagree with this design, but I think it'll take some time for me to get used to it. There are some minor things that break my expectations, one of them being swiping down on an app does not bring it into the foreground, it instead hides the drawer to reveal the app that was previously open. Another fun one is the bottom bar that indicates that you swipe up to open the drawer again. This is pretty handy, and they've included a little button in the corner to toggle the keyboard, which may be necessary for less touch-compatible applications. An oversight is that the keyboard button isn't available in the drawer itself, so if you search for applications, you can't close the keyboard again. </a> The notifications panel, which features familiar Android style quick options, including sliders for volume and brightness.</figcaption> </figure> </a> Screenshot of the same notifications panel, but with the audio selector expanded to show options for changing audio devices.</figcaption> </figure> The notifications / quick options panel also looks pretty slick and heavily inspired by Android. Quick settings are a must-have for phones in my opinion, and it's a nice touch to include the GNOME audio device selector. Audio device selection can be finicky on Android - on my previous Lineage OS phone it was easily accessible by expanding the volume indicator, but Samsung in their infinite wisdom decided to implement this as a separate application, opened from the quick panel. Just that second it takes to cold launch irks me. There are some fairly simple oversights as of writing. One such oversight is long pressing on the quick options to open the related menu in settings. However, doing so 1) doesn't hide the app drawer if it's open already, and 2) doesn't focus the settings application if it's not already focused. While the audio device selector is super handy, it doesn't reset when you close the notification panel, so it will be open when you slide the notification panel down again. I imagine most users won't like having its state persisted like this. </a> The WiFi hotspot GNOME popup, which works okay in Phosh.</figcaption> </figure> </a> The WiFi connection settings GNOME popup, which does not work well.</figcaption> </figure> Popups are something Phosh hasn't quite worked out. Of course, these are designed for a desktop, where having windows overlap is commonplace. But this model doesn't work for a window manager which expects only one window to be visible at a time. How this currently works is that the popup simply displays on top of the current window, and it's just luck whether that particular menu will fit on the screen. The popup displays as its own entry in the "open app list", so you can swipe it away to close it if you can't reach a close button. The first issue is the fundamental behaviour of popups. If we consider Android, a popup menu will take over the whole screen, dimming what's in the background. To close it, one can usually tap outside of the popup, or use the back button / gesture. That back button is missing in Phosh; the bottom bar only has a swipe up to access the app drawer and window list, and a keyboard button in the bottom right. The issue of overflowing menus is a much larger one to tackle. A lot of menus are designed with desktop aspect ratios in mind, and certainly not for 200% resolution scaling on such small screens. A lot of effort has been put into making some GNOME applications - particularly the settings - reactive to thinner screens. In some places like the display settings, this reactivity is buggy, in this case the menu seems to be fighting its own layout and jumping around every repaint. Other menus have been left completely neglected, such as the WiFi connection settings. This one uses a tabbed layout to separate the separate sub-menus. A generic solution might perhaps be automatically replacing this tabbed menu with a vertical sub-menu system, where each tab is its own menu entry. Similarly, instead of a sub-menu, the tab entries could expand vertically to reveal its sub-menu. But another problem with this connection settings menu is the multi-column layout used to spread options horizontally. Making this layout dynamic won't be trivial. I've never worked with GTK to this level, let alone GNOME in particular, so I can't say just how easy these things would be to implement. But I'll give it a try anyway. A lot of people have put a lot of hard work into this project already, and I'd like to help if possible. Given how the window system is the very fundamental to the user experience of a touchscreen device, getting it right will do wonders for the usability of mobile GNU/Linux. Expectations have been set by the Android and iOS operating systems, which have both seen decades of development in touchscreen UX. While there are many valid complaints to make about Apple and Google's design choices in places, it's safe to say that they are both setting the bar, and it's been set quite high. Ultimately, we've all been using these phone operating for years now, so we're all quite accustomed to their patterns. It's probably best to start with the familiar before making opportunistic improvements where appropriate. Epiphany</h3> The app initially refused to launch, dying immediately. Running it through the terminal, it complained about not being able to write to ~/.local/state</code>. For some reason, (possibly from nixos-install?) this was owned by root. I chown -R</code>ed the directory to resolve that. </a> My website rendered in Epiphany. Please ignore the broken icons, my icon pack seems to be missing some.</figcaption> </figure> Epiphany looks notably better than the PureOS version. The UI looks very much inspired by Chrome on Android (though I haven't used that in forever). It even has Firefox sync support! I tried logging in, but I got a keyring setup prompt, followed by some error relating to the keyring. Funnily enough, the second password prompt to confirm the password had the "Store passwords unencrypted?" text. The GNOME swipe from the left to go back gesture comes quite handy here, but it does trigger accidentally while browsing quite often. It should probably be restricted to a thin trigger zone on the side in these sorts of applications. Annoyingly swipe gestures are used inconsistently, e.g. in the tab list menu, I'd expect to be able to swipe away tabs, but only the close button works. I'd also prefer if the URL bar was on the bottom, as is the modern trend in browsers. That is much more accessible to my fingers, given how tall modern phones are. I'd like to go to a new site without awkwardly stretching my hand, or using a second one. Camera</h3> Mobile NixOS installs megapixels</a> by default. Unfortunately this app also crashes on launch, but it's a less transient error. It complains about not being able to find a config file with the phone's model in the name. Clearly the app is specifically configured for each device, and you can find each config file here</a>. I'll have to do some tinkering and create one for the Librem 5, then I can submit a patch for it. Other apps</h3> </a> The GNOME Calendar app doesn't fare too well, with the right side cropped.</figcaption> </figure> </a> Similarly, the Activity Monitor app isn't adjusted for mobile at all, and most of the window renders off screen.</figcaption> </figure> Apps like Calendar, Activity Monitor, and the standard GNOME text editor are curiously displayed as non-mobile applications, and look different than on PureOS. Compared to this showcase</a> back in 2022, on NixOS these apps are notably missing a mobile-style bottom navigation bar. That suggests to me that Librem have forked these apps to adjust the UI for phone use. I'll have to look at packaging these for NixOS. </a> The GNOME clocks application looks pretty similar to PureOS, and is very usable on the Librem 5</figcaption> </figure> The Alarm application also has a good mobile interface, but unfortunately it doesn't work well as an alarm clock. It doesn't appear to set the RTC wakeup alarm, so the phone doesn't wake up to sound the alarm. There's an open ticket for this</a> which is 3 years old as of writing. NixOS problems</h3> A weird issue I've been grappling with is some lingering garbage collection roots pointing to /proc</code> for processes that no longer exist, I suspect from previous boots. Nix doesn't seem to clean those up no matter what I do, and I've yet to work out how to fix this. If unresolved, it's a big issue on the phone given the limited storage space (32GB), of which the Nix store is taking up 7GB. There's effectively two full systems in there, one of which is completely unnecessary. Next steps</h2> I'd like to start by making it easier for others to get to this point. I didn't quite know how easy this was going to be from the start, and I certainly complicated things for myself with the nixos-anywhere tangent. However, I think there are some opportunistic improvements to documentation, particularly: How to override the Nix build directory with TMPDIR</code></li> Getting started with Mobile NixOS The current Device Porting Guide page</a> is pretty bare-bones.</li> How to first flash the phone. This issue</a> suggests using the Mobile NixOS installer image by booting it from the SD card, but as far as I can tell that's not possible on the Librem 5.</li> </ul> </li> </ul> After that I'd like to try adding support for the Librem 5 in the Mobile NixOS project. It seems like a more appropriate home for it, instead of the nixos-hardware repository. I don't know quite how it'll work to reproduce Sophie's work elsewhere. I also need to work out some more things in Mobile NixOS, such as how kernel upgrades and recovery works. Surely there's a smarter system than re-writing the boot partition each time. After that, I think I'll try resolving some of the problems I described in the first impressions. Starting with the easier ones, of course. I hope I won't have to do much compiling, or figure out a faster method. Emulated compilation is painfully slow, and a compilation cache doesn't always help. Anyway, we'll see how it goes. I hope to make some meaningful contributions to bring the dream of a feasible GNU/Linux phone a reality. A Journey into Nix and NixOS Unknown — Fri, 21 Oct 2022 00:00:00 +0000 Before I started this journey, I ran this website and a few personal services on the cheapest VPS OVH offer</a>, but the limited disk space has frequently given be trouble. 20GB isn't a lot to work with when you're using it to synchronise and stash personal files. I planned to take cancel its upcoming renewal, and instead migrate to another VPS provider that provides a bit better value for money. However, this introduces the problem of manually migrating and setting things up again. Naturally, having done most of the setup over a year ago and I'd forgotten most of it. I wasn't keen on repeating that mistake: I wanted a proper automated deployment method. One that doesn't rely on me doing everything manually, or rely on all the knowledge staying in my head. While Ansible and similar are the "industry-standard" tools for doing this. I've seen Ansible used at a previous job, and witnessed how slow it can be, and now messy it can get with its piles of YAML. Needless to say, I wasn't left with the best impressions, and wasn't too keen on it for personal use. I'd came across Nix online through articles and discussions, including Xe Iaso's blogs and talks</a>, and the Self Hosted podcast</a>. Its approach sounded novel and interested me, so I took the server migration as an opportunity to experiment with it. I wrote most of this article as I went along, and it chronicles most of my trial-and-error in creating a NixOS. It's mostly a stream of conciousness, serving primarily as a record of what I struggled with and thoughts that I had along the way. In a subsequent article, I'll write up my perspective on Nix and NixOS and what I'd like to see from it in the future. That one will hopefully be less ramble-y and more interesting to the average reader. It's been a long road, I've been working on it on and off for months now, starting in May 2022 and continuing until just before this article was published. Thankfully I consider this experiment a success, my data and services have all been migrated to the new server before my old VPS was up for renewal renewal, as I'd planned. It's been quite messy and frustrated at times, but I'm quite happy with how things turned out. I think all this work is a "once and for all" effort to ensure I don't have to manually set up my server again. I'm also reasonably happy with Nix, and I think I'll be using it more in the future. If you're interested in viewing the end result (i.e. my NixOS configuration), you can find it in my configuration repository</a>. Preparatory reading</h2> As with most things, I initially started with some web searches, and came across How to Learn Nix</a>, which I ended up reading through quite a lot of. I additional came across Nix Pills</a>, which I think I briefly skimmed, and the Nix Wiki</a>, which I initially referred to frequently while getting started. Installing on my Arch computer</h2> Fairly straightforward, it's available in Arch's community repository, so I installed it through pacman. After hitting some errors, I followed the wiki's guide and added the nixpkgs-unstable channel. Then looked at managing user config files. I found home-manager but sounded like way more than I need - I want to keep my dotfiles for portability when I can't use Nix, I just need a little Nix expression for installing the few config files that need to live outside of ~/.config as symlinks. A user Nix config for installing applications would be neat, but I decided to postpone until I'm further along. I decided to make my first goal reproducing my webserver configuration with Nix. So on to setting up a playground environment before investing in another webserver. Raspberry Pi</h2> I have a Raspberry Pi free for some experimentation, so I thought I'd use it as a testbed in place of a real VPS. The wiki article suggests downloading and writing the SD card, but it's a graphical installer. I don't have a micro HDMI for the Pi so I want a headless installation with SSH (which is what I used with Arch on ARM). I assume I need to build my own image as suggested in the wiki, including my ssh key. Initially Followed the wiki article, using cross-compilation. { ... }: { nixpkgs.crossSystem.system = "aarch64-linux"; imports = [ <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix> ]; users.extraUsers.root.openssh.authorizedKeys.keys = [ "ssh-rsa ...." ]; } </code></pre> $ nix-build --system aarch64-linux --keep-failed --store /data/nix-raspberrypi '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config="$HOME"/.config/nix/raspberrypi-sdcard.nix -I nixpkgs=channel:nixos-21.11-aarch64 </code></pre> What does the command mean? Strangely it wanted to rebuild a lot of applications. Why isn't it downloading everything from the cache? First roadblock was an error in liburing, but hmm this appears to be building fine on hydra. I find the --keep-failed</code> option to actually inspect what's wrong. Some the post install script referred to a nonexistent file. Tried adding the nixos-21.11 and nixpkgs-21.11-aarch64 channels - no change. Found the --dry-run</code> option - why are the hashes on hydra different to my nix-build? (Was it because I didn't pass --pure</code>?). I followed some of my previous reading material and managed to override the liburing derivation using overrideAttrs</code>: { ... }: { nixpkgs.crossSystem.system = "aarch64-linux"; imports = [ <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix> ]; nixpkgs.overlays = [ / (self: super: { liburing = super.liburing.overrideAttrs (attrs: { postInstall = '' # Copy the examples into $bin. Most reverse dependency of this package should # reference only the $out output mkdir -p $bin/bin cp ./examples/io_uring-cp examples/io_uring-test $bin/bin cp ./examples/link-cp $bin/bin/io_uring-link-cp ''; }); }) / ]; users.extraUsers.root.openssh.authorizedKeys.keys = [ "ssh-rsa ...." ]; } </code></pre> Also try the undocumented --store</code> to override the store because my root drive is running out of space. This causes issues later, so I clear up some space and use the default /nix/store</code> instead More building... Why am I building the Linux kernel? And the AMD drivers and Nouveau? I don't need those. I hit "no space on device" errors, my tmpfs is running out of space building all of this stuff. Find some GitHub issues that mention remounting /tmp</code> and resizing to 14GB. Fine, I do that. (Later someone mentions that it actually uses $TMPFS</code> rather than hardcoding /tmp</code>, so I could have used a different filesystem.) Find the nix-store --read-log</code> option, which is quite neat. Before finding it, I ran builds several times over after losing the tmux scrollback buffer (and later increasing the tmux scrollback limit). More building... Make: error 2 during linking(?) At this point I was lurking in the NixOS on ARM channel, someone mentioned to someone else that cross compilation isn't reliable. Suggests the qemu-user method. Fine, time to try that. I install qemu-user</code> and the binfmt</code> packages from the AUR, add this to /etc/nix/nix.conf</code>: extra-sandbox-paths = /usr/bin/qemu-aarch64-static extra-platforms = aarch64-linux </code></pre> Still get some errors. I try overriding nixpkgs.system</code> and system</code>, no dice. Apparently --argstr system aarch64-linux</code> isn't working right now. Find another undocumented option, --system</code>, which does appear to work. Later I find out (from the chat?) the distributed SD card image does support SSH, you just need to create /root/.ssh/authorized_keys</code> on the flashed system. So that whole exercise was a waste of time. I flash the SD card, plug Pi and Ethernet in, no green light, but lights on Ethernet. Pi debugging ensues, replugging the SD card before Ethernet eventually works. Pi shows up on the router's web UI, but isn't assigned an IP address. DHCPD issue? I have no idea, and I'm not sure how to debug. I give up, maybe I should try a VM, which should be simpler, right? Building a QEMU image</h1> I find a couple of guides for making VMs with Nix: https://gist.github.com/tarnacious/f9674436fff0efeb4bb6585c79a3b9ff</li> https://gist.github.com/573/c1d73a4fd04b8f8ca63885393856f9ea</li> </ul> I also find a link to the VM Nix module</a>, which as all the available options. I ideally want no graphical window, just a shell. Why isn't nix-build -A vm '<nixpkgs/nixos>' --arg configuration "{ imports = [ <nixpkgs/nixos/modules/virtualisation/build-vm.nix> ]; }" -I nixos-config=vm.nix</code> working? Works if I put everything in vm.nix</code> and add the build-vm.nix</code> import. Probably the configuration</code> argument being ignored. VM template: { pkgs, ... }: { imports = [ <nixpkgs/nixos/modules/profiles/qemu-guest.nix> <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix> ]; config = { system.stateVersion = "22.05"; fileSystems."/" = { device = "/dev/disk/by-label/nixos"; fsType = "ext4"; autoResize = true; }; boot = { growPartition = true; kernelParams = [ "console=ttyS0" ]; loader.grub.device = "/dev/vda"; loader.timeout = 0; }; users.extraUsers.root.password = ""; services.getty.autologinUser = "root"; networking.hostName = "vm"; virtualisation = { cores = 4; graphics = false; qemu.options = [ "-serial mon:stdio" ]; # Forward any needed ports forwardPorts = [ { from = "host"; host.port = 8080; guest.port = 80; } ]; }; nix.extraOptions = " extra-experimental-features = nix-command flakes "; }; } </code></pre> Runs successfully with: $ nix-build '<nixpkgs/nixos>' -A vm --arg configuration ./vm.nix $ ./result/bin/run-vm-vm </code></pre> I'm not sure how to break down the nix-build</code> call. Include <nixpkgs/nixos</code> and vm.nix</code>, evaluate the expression vm</code>? What is the vm</code> attribute? Where does it come from? </blockquote> Writing a Nix module for my webserver</h1> Nextcloud</h2> I first find the wiki page</a>, but it's pretty barren. It doesn't even link to manual page. I follow the next search result and find the manual section. Setup is mostly straight-forward, most of the time spent trying to create these secret files on the disk. Required a bit more exploration of the Nix language. I initially think I could manage them by overriding options/attribtues when calling nix-build</code>. However the nix-build</code> switch --option</code> appears to refer to Nix options, i.e. things that go in /etc/nix/nix.conf</code> --arg</code> and --argstr</code> appear to be function arguments, i.e. things specified in { arg }:</code> at the start of a module Is there a point in exploring this approach, if it'll be in plaintext in the store anyway? In the next section, I look into NixOps and if/how it handles secrets sensibly. Selecting Nextcloud apps is a bit janky, I was expecting it to just need the app name, but it wants you to manually specify the download URL, hash, and all that jazz manually. I assume its for reproducibility, and no one's created packages for Nextcloud apps yet. Dealing with secrets</h2> Sorting this out probably going to be unavoidable if I want to use Nix "properly", this is, running one command to completely reproduce a system with virtually zero manual work required. I find the wiki article on the subject</a> Lots of options - none of them ideal and zero extra effort. Proper secret management within Nix and the Nix store has been on the back-burner for a long time: https://github.com/NixOS/nix/issues/8. My ideal: hook into pass, my password manager, to retrieve my secrets. Let me specify the required ones in my configuration, and deploy them when I'm running NixOps or whatever, with appropriate (and configurable) file permissions. Let them be persistent so I don't have to redeploy them every time, and clean up older ones if I remove them from the config. NixOps</h2> Okay, the default NixOps key system looks fine. You can override where keys are saved so I can create them in /secrets</code> or something. I'm thinking to use NixOps to deploy to real servers anyway, so I'm gonna have to learn it at some point. Looks like it has some features for delaying/restarting systemd units when secrets change or are made available, which is handy. What is up with the NixOps manual on nixos.org</a>? It has very little info, is lacking context on how to write NixOps configuration files and load commands. Proper manual appears to be on releases.nixos.org</a> Can I override the SSH identity file used or do I really need to adjust my ~/.ssh/config? I type in the host name as I would if I were running ssh <host></code>, but for some reason not all SSH options are picked up. E.g. I need to set deployment.targetPort</code>, otherwise NixOps tries to connect to port 22. $ nixops deloy</code> is building stuff - is it using the version of nixpkgs I'm running NixOps from? Do I need to create another deployment to reload the NixOps network file? No, it looks okay in the end. But hits some errors at the end of the deploy, I assume running post-update GRUB hooks? vm> updating GRUB 2 menu... vm> installing the GRUB 2 boot loader on /dev/vda... vm> Installing for i386-pc platform. vm> /nix/store/4bmcgmin11ixrn2ij2jjinlddcxql1g6-grub-2.06/sbin/grub-install: warning: File system `ext2' doesn't support embedding. vm> /nix/store/4bmcgmin11ixrn2ij2jjinlddcxql1g6-grub-2.06/sbin/grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged.. vm> /nix/store/4bmcgmin11ixrn2ij2jjinlddcxql1g6-grub-2.06/sbin/grub-install: error: will not proceed with blocklists. vm> /nix/store/lc7cvzrgsxks87g3jnjbg8pg2pqn15vh-install-grub.pl: installation of GRUB on /dev/vda failed: No such file or directory vm> error: Traceback (most recent call last): File "/nix/store/4zh7crx1r2sizvpyb1c9h109yfimlzn2-python3-3.9.12-env/lib/python3.9/site-packages/nixops/deployment.py", line 893, in worker raise Exception( Exception: unable to activate new configuration (exit code 1) </code></pre> Why is it installing for i386? My VM is x86_64. Why is detecting the filesystem as ext2? /dev/vda</code> is ext4. I assume there are errors from updating within the VM itself, and probably not NixOps' fault. It just happened to update some packages. Further research: grub-install</code> is apparently trying to update an MBR partition with UEFI stuff? Try to force the VM to use UEFI with GRUB by inspecting the qemu module</a>. I find a related option, and try setting virtualisation.useEFIBoot = true</code>: get the following error: qemu-kvm: -drive if=pflash,format=raw,unit=1,file=: A block device must be specified for "file" </code></pre> Further inspection: there is a dependency between useEFIBoot</code> and useBootLoader</code> (the latter defaults to false</code>) I take a brief detour to enable KVM in my BIOS. With the new changes, the VM just boots into the UEFI shell, instead of directly to the system. Back to the normal VM. It's quicker now due to the KVM modules loaded, but of course back to the same NixOps error. Maybe I should create VMs through NixOps instead, which will be better supported? How about a quick detour and using sops-nix</a> instead? (In hindsight, this detour was not quick at all) sops-nix and Flakes</h2> Again, not ideal because I'd prefer to just use passwords stored in pass</code>. But I understand why sops is preferable when it comes to Nix - the encrypted secrets can be used as an input to the system configuration. For now it seems like one of the easier options to go for. I like that it apparently hooks into nixos-rebuild</code> and the like. Quick question, how do I use nixos-rebuild</code> when I'm not on NixOS? Turns out it's packaged in nixpkgs, duh</a>. I run it with $ nix-shell -p nixos-rebuild</code> sops-nix requires the target host's SSH key which is rather annoying, since it means having to deploy after creating a machine. And some manual work involved in extracting the SSH public key of new machines. Setup guide is rather convoluted and complex, due to multiple encryption methods: using age keys, converting GPG or ssh keys to age keys. Try to copy sops-nix example, so I guess I'm trying out flakes now: { inputs = { nixpkgs.url = "nixpkgs/nixpkgs-unstable"; sops-nix.url = "github:Mic92/sops-nix"; sops-nix.inputs.nixpkgs.follows = "nixpkgs"; # use the same nixpkgs from this flake's input, this seems hacky }; outputs = { self, nixpkgs, sops-nix }: { nixosConfigurations."vm" = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; modules = [ ./vm.nix sops-nix.nixosModules.sops ]; }; }; } </code></pre> Try to deploy with $ nixos-rebuild switch --target-host vm --flake '.#vm'</code> error: getting status of '/nix/store/ppm1c1rch1p6w260kpfxma6kww8wkfdf-source/nix/flake.nix': No such file or directory </code></pre> Okay, turns out I need to git add flake.nix</code> for some reason. error: cannot look up '<nixpkgs/nixos/modules/profiles/qemu-guest.nix>' in pure evaluation mode (use '--impure' to override) </code></pre> I guess I need to explicitly add the nixpkgs flake? Do flakes change the way imports work? How to I translate a <nixpkgs/...></code> import into an expression? Now I try to rebuild my VM, but now I'm getting a new error: error: The option `sops' does not exist. Definition values: - In `/home/avery/.config/nix/vm.nix': { defaultSopsFile = /home/avery/.config/nix/secrets/common.yaml; secrets = { "nextcloud/admin" = { }; "nextcloud/database" = { }; </code></pre> Durr, this is because I'm still running nix-build</code> but I'm using the sops option in my configuration. I assume I need to convert this to the new nix build</code> command to make use of Flakes? Let's give this a shot. The VM module</a> creates system.build.vm</code>, which is probably what I'm currently invoking with nix-build</code>. The nix build --help</code> has the following example: Build a NixOS system configuration from a flake, and make a profile point to the result: </blockquote> # nix build --profile /nix/var/nix/profiles/system \ ~/my-configurations#nixosConfigurations.machine.config.system.build.toplevel </code></pre> That looks pretty similar! Surely I can change ~/my-configurations</code> and machine</code> appropriately, replace system.build.toplevel</code> with system.build.vm</code>, and that's my new build command? $ nix build .#nixosConfigurations.vm.config.system.build.vm warning: Git tree '/home/avery/.config' is dirty error: cannot look up '<nixpkgs/nixos/modules/profiles/qemu-guest.nix>' in pure evaluation mode (use '--impure' to override) at /nix/store/h7ncs2xdj5nmfwfrgnhxv4wr0q6v6fqd-source/nix/vm.nix:8:5: 7| imports = [ 8| <nixpkgs/nixos/modules/profiles/qemu-guest.nix> | ^ 9| <nixpkgs/nixos/modules/virtualisation/qemu-vm.nix> (use '--show-trace' to show detailed location information) </code></pre> Progress? Now, how to import these nixpkgs modules... they're not exposed through the nixpkgs flake, are they? The Flakes wiki page has a Making your evaluations pure</a> section, this suggests I need to somehow download these nixpkgs modules that I'm trying to import? Can I import a module within a flake's repository directly? Surely the entire repository is already in the nix store anyway? Yes! I turns out that I've had a incorrect presumption. I don't grasp the reason but even in the case of being a flake input inputs.home-manager will still return the store path of the home-manager flake. Which means I can still use it as an ordinary non-flake input That's Great A comment</a> by ykis-0-0 on reddit.com/r/NixOS </cite> </blockquote> This hint led me to add a trace to within the outputs</code> function in flake.nix</code>, and lo and behold, it just points to the store path for nixpkgs! { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; sops-nix.url = "github:Mic92/sops-nix"; sops-nix.inputs.nixpkgs.follows = "nixpkgs"; # nasty hack to use the same nixpkgs from this flake's input }; outputs = { self, nixpkgs, sops-nix }: { nixosConfigurations."vm" = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; modules = [ (nixpkgs + "/nixos/modules/profiles/qemu-guest.nix") (nixpkgs + "/nixos/modules/virtualisation/qemu-vm.nix") ./vm.nix sops-nix.nixosModules.sops ]; }; }; } </code></pre> Not ideal, since now the import has moved outside of the vm.nix</code> module which actually needs these imports. Question: Can I somehow move this import to inside the module? (I found out the answer much later: Yes! Using specialArgs</code>) Getting another error now: $ nix build .#nixosConfigurations.vm.config.system.build.vm warning: Git tree '/home/avery/.config' is dirty error: getting status of '/nix/store/2awpx4cw6r4y4zxd3rnx93xq4h2mb9l3-source/nix/common.nix': No such file or directory (use '--show-trace' to show detailed location information) </code></pre> If I explore this store, common.nix</code> is indeed missing, but most of the files are there! Also, this store contains a snapshot of my entire Git repository, most of which isn't needed by the Nix stuff. But that's suspicious - is it because common.nix</code> isn't staged yet? Why indeed, it isn't. Is this what the warning is about? Maybe that warning should also specify that only staged files will be used during the build, and especially check special files like flake.nix</code>, it's confusing for builds to immediately fail with an error that doesn't make sense unless you know that it's copying Git-tracked files to the Nix store. I have everything ignored by default in my Git repository (because it lives in ~/.config</code>, which many applications dump files into), so I hit this problem repeatedly. Finally, success! I've now converted to flakes, time to test out sops-nix again. I can see sops-nix trying to do something at the start, but since I've wiped my VM its ssh-key has changed: setting up secrets... sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint cb3e5e4ce4278da9a1bb7beb7ff790665581e7ad /nix/store/6a7866mckid9n3s8bvanp5ag5fk2awyl-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/p2inyam4wdxgljisqx035y1i8f85npi5-common.yaml': Error getting data key: 0 successful groups required, got 0 Activation script snippet 'setupSecrets' failed (1) </code></pre> Time to redo part of the sops-nix setup. I get why it's done this way - using the host's SSH key is a neat idea - but I don't like that it's a manual task that needs to be done after initially booting a system. Maybe it's more of a problem with these throwaway VMs that I'm using. Updated the age key for the VM, then ran sops updatekeys <file></code>. New error on boot: setting up secrets... sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint cb3e5e4ce4278da9a1bb7beb7ff790665581e7ad /nix/store/6a7866mckid9n3s8bvanp5ag5fk2awyl-sops-install-secrets-0.0.1/bin/sops-install-secrets: Failed to decrypt '/nix/store/zl4yjsw14if8wqrblnf1lk50y01zvl6w-common.yaml': Error getting data key: 0 successful groups required, got 0 Activation script snippet 'setupSecrets' failed (1) </code></pre> Is sops-nix trying to use the host's RSA key? I thought the age method only supported the ed25519 key? My mistake: the sops-nix guide says to set the following: sops.age.sshKeyPaths = [ "/etc/ssh/ssh_host_ed25519_key" ]; </code></pre> Also, I was accidentally using my actual host's SSH keys because I ran $ ssh-keyscan localhost -p 2222</code> instead of $ ssh-keyscan -p 2222 localhost</code>. That one's on ssh-keyscan</code>. On top of that, ssh-keyscan</code> doesn't detect the ed25519 keys on the VM for some reason, so I had to copy the public key manually and pipe it through to ssh-to-age</code>. One last $ age updatekeys ...</code>, and now there are secrets! setting up secrets... sops-install-secrets: Imported /etc/ssh/ssh_host_rsa_key with fingerprint cb3e5e4ce4278da9a1bb7beb7ff790665581e7ad ... $ ll /run/secrets/ total 4 drwxr-x--x 2 root keys 0 May 29 12:59 nextcloud -r-------- 1 root root 30 May 29 12:59 smtp </code></pre> Went back and amended Nextcloud configuration to use sops-nix instead of Nix options. Finally, time to get back to the task at hand. Quick diversion: setting up a Nix formatter</h2> Initially running nix fmt</code> gives a somewhat helpful (but not exactly user-friendly) error: warning: Git tree '/home/avery/.config' is dirty error: flake 'git+file:///home/avery/.config?dir=nix' does not provide attribute 'formatter.x86_64-linux' </code></pre> It should probably indicate to check nix fmt --help</code>, because that page has a few examples to copy-paste from. I chose nixpkgs-fmt</code> and added this to the output</code> set in my flake.nix</code> as directed: { output = { formatter.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.nixpkgs-fmt; }; } </code></pre> Making a package: systemd-failmsg</h2> I use systemd-failmsg</a> to get emails when a service fails on my webserver. I definitely want it on my new server, but it's as-yet unpackaged. So I guess I need to create a proper Nix package for it? Let's see this wiki page</a> It basically says "look for something similar in nixpkgs and copy that". Which isn't ideal. Some basic templates would be nice. Okay, I've copied something that looks reasonable: { stdenv, lib, fetchFromGithub, ... }: stdenv.mkDerivation rec { pname = "systemd-failmsg"; version = "1.3"; src = fetchFromGithub { owner = "dino-"; repo = pname; rev = "v${version}"; sha256 = "7a2a6cc9311f1370b1c295d3cf0428604e03ced08bb894e073cd58209f5ef537"; }; meta = with lib; { homepage = "https://github.com/dino-/systemd-failmsg"; description = "systemd toplevel override which sends emails alerts when systemd services fail"; license = licenses.isc; platforms = platforms.linux; }; } </code></pre> How to I build it? $ nix build --file ...</code> gives me: error: anonymous function at /home/avery/.config/nix/pkgs/systemd-failmessage.nix:1:1 called without required argument 'stdenv' </code></pre> The wiki article doesn't go into detail on using $ nix build</code> to build a package. Most of the existing documentation uses nix-build</code> instead, which obviously doesn't make use of flakes. Okay, a different approach: specifying the package in flake.nix</code>. The wiki page has very little detail on how to use output.packages</code> other than specifying that it's used by $ nix build .#<package></code>. nixpkgs only uses legacyPackages</code>, so I have to search elsewhere. I looked at the Flake RFC</a>, in which a comment mentions the callPackage</code> pattern used by nixpkgs for its legacy packages, but within the output</code> function of my flake nixpkgs</code> just refers to the store path. It took me a bunch of searching until I found this blog</a> which mentions actually importing nixpkgs</code> to provide the classic pkgs</code> input. Makes sense once it's done in an example! { # ... outputs = { self, nixpkgs, sops-nix }: let forAllSystems = f: nixpkgs.lib.genAttrs nixpkgs.lib.systems.flakeExposed (system: f system); in { packages = forAllSystems (system: import ./pkgs { pkgs = import nixpkgs { inherit system; }; } ); nixosConfigurations."vm" = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; modules = [ (nixpkgs + "/nixos/modules/profiles/qemu-guest.nix") (nixpkgs + "/nixos/modules/virtualisation/qemu-vm.nix") ./vm.nix ./common.nix sops-nix.nixosModules.sops ]; }; formatter.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.nixpkgs-fmt; }; } </code></pre> I also copied the forAllSystems</code> function used by nixpkgs' flake.nix</code> so my packages are cross-platform by default. Tried $ nix build .#systemd-failmsg</code>, Encountered a very helpful error message! error: You meant fetchFromGitHub, with a capital H </code></pre> Given how specific this error is ("with a capital H"), this is probably hard-coded message for this specific misspelling of a function. However, it's an example of what error messages should look like in practice. The most related section of the NixOS manual appears to be Adding custom packages</a>, which dutifully links to the more details nixpkgs manual</a> which actually covers packaging in detail. My skim-reading caused me to skip over this link - I instead searched for mkDerviation</code> in the NixOS manual but found very little. Nevertheless, I was able to piece the puzzle together from the NixOS manual alone, working out my roadblocks: Implementing the installPhase</code> attribute</li> Using bash</code> as an input to run the package's install script (otherwise the script would result in "/bin/bash: bad interpreter: No such file or directory"</li> Using the $out</code> variable as the install prefix (instead of simply out</code>) The error message for this is a bit vague: error: builder for '/nix/store/4k49hl64mvwijpxc5v53szqv5vjzkhw6-systemd-failmsg-1.3.drv' failed to produce output path for output 'out' at /nix/store/... </li> </ul> </li> </ul> However, after these issues I find everything as expected under result/</code>. I'm pleasantly surprised that Nix automatically substitutes the shebang in a script which is part of the package - #!/bin/bash</code> is replaced with #!/nix/store/.../bash</code>. Excellent, that spares me from some tedious patching. I can now search for my package and even show its derivation, nifty! $ nix search . systemd-failmsg packages.x86_64-linux.systemd-failmsg (1.3) systemd toplevel override which sends emails alerts when systemd services fail $ nix show-derviation .#systemd-failmsg { "/nix/store/l14p85r1cn2fix4sisr6c1cfmgzm4x3p-systemd-failmsg-1.3.drv": { "outputs": { "out": { "path": "/nix/store/bxj557f73c5k8ayirj30qlh4c4lvwc5w-systemd-failmsg-1.3" } }, "inputSrcs": [ "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh" ], "inputDrvs": { "/nix/store/3i6jqp61ra3031kjs1jlrmmqd2jyixd6-source.drv": [ "out" ], "/nix/store/42pr7zqjf0y29v19q1wxn6hs5gdl5car-bash-5.1-p16.drv": [ "dev", "out" ], "/nix/store/ddmyhp06jqy8bxj715zwsmbcnzvx8iax-stdenv-linux.drv": [ "out" ] }, "system": "x86_64-linux", "builder": "/nix/store/0d3wgx8x6dxdb2cpnq105z23hah07z7l-bash-5.1-p16/bin/bash", "args": [ "-e", "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh" ], "env": { "buildInputs": "", "builder": "/nix/store/0d3wgx8x6dxdb2cpnq105z23hah07z7l-bash-5.1-p16/bin/bash", "configureFlags": "", "depsBuildBuild": "", "depsBuildBuildPropagated": "", "depsBuildTarget": "", "depsBuildTargetPropagated": "", "depsHostHost": "", "depsHostHostPropagated": "", "depsTargetTarget": "", "depsTargetTargetPropagated": "", "doCheck": "", "doInstallCheck": "", "installPhase": "PREFIX=$out bash install.sh\n", "name": "systemd-failmsg-1.3", "nativeBuildInputs": "/nix/store/qcalxj277ld4jiklmd2lzx6gkcvkc67k-bash-5.1-p16-dev", "out": "/nix/store/bxj557f73c5k8ayirj30qlh4c4lvwc5w-systemd-failmsg-1.3", "outputs": "out", "patches": "", "pname": "systemd-failmsg", "propagatedBuildInputs": "", "propagatedNativeBuildInputs": "", "src": "/nix/store/ig1jp44jq5vy1lmdkm6ilimhq96v6157-source", "stdenv": "/nix/store/28hqpbwpzvpff7ldbhxdhzcpdc34lgsa-stdenv-linux", "strictDeps": "", "system": "x86_64-linux", "version": "1.3" } } } </code></pre> And finally, I can add my package to the machine's configuration. The question is... how? Again, this would be easier if I could pass stuff from flake.nix</code> into child modules. Doing a bit more research, it is indeed possible, with the specialArgs</code> attribute, which is a set that is merged with the usual parameters that are passed to imported modules. I figured out that the rec</code> keyword means "recursive" from the NixOS manual, and it allows references to other attributes within an attribute set, so I can include my flake's packages as an argument to the other modules: { outputs = { self, nixpkgs, sops-nix }@inputs: let forAllSystems = f: nixpkgs.lib.genAttrs nixpkgs.lib.systems.flakeExposed (system: f system); { packages = forAllSystems (system: import ./pkgs { pkgs = import nixpkgs { inherit system; }; }); nixosConfigurations."vm" = nixpkgs.lib.nixosSystem rec { system = "x86_64-linux"; # Merge all flake inputs with the packages specialArgs = inputs // { mypkgs = packages.${system}; }; modules = [ ./vm.nix ]; }; }; } </code></pre> And add it to my system's configuration: { mypkgs, ... }: { imports = [ /* ... / ]; environment.systemPackages = with mypkgs; [ systemd-failmsg ]; } </code></pre> Here's the tree of the package: $ tree /nix/store/...-systemd-failmsg-1.3/ ├── bin │ └── failmsg.sh ├── lib │ └── systemd │ └── system │ ├── failmsg@.service │ ├── failmsg@.service.d │ │ └── toplevel-override.conf │ └── service.d │ └── toplevel-override.conf └── share └── systemd-failmsg ├── always-fails.service └── doc ├── LICENSE └── README.md </code></pre> I can rebuild and run the VM successfully, but the systemd services aren't available. I do see the script in bin/failmsg.sh</code>, but nothing else. Instead of using the provided install script, I'll try to do things the NixOS way. My initial alternative approach involved creating a Nix module for systemd-failmsg</code>, following the pattern used by other packages/modules that provide systemd services, e.g. nginx</a>. I tried to create the systemd unit file with systemd.service.<service> = </code>, and the number of config files for the overrides using environment.etc</code>. This, however, produced a number of "Permission denied" errors during building the env</code> package. Seemingly the etc/systemd/system</code> path was already owned by root at the respective part of the build, was it already locked down and considered read-only? I didn't have any luck searching for this issue, but I assume this is an un(der)documented special case for systemd and other core system packages. I eventually discovered the correct solution while browsing existing examples and issues. I was first directed to /run/current-system/sw</code>, where I found my service under /lib/systemd/system</code> as I expected. This explains why /lib</code> doesn't exist - it's instead stashed away in here. It also shows that Nix does intelligently install the systemd service and the configuration files. The service did not however show up in systemctl list-units</code>, suggesting I was missing another step. This step was adding the package to the systemd.packages</code> list, which according to the manual</a> "enables" the service. After doing this, my service and its systemd overrides show up under /etc/systemd</code>, and are all picked up correctly. In the end I was left with this fairly short module: { config, lib, mypkgs, ... }: with lib; let cfg = config.services.systemd-failmsg; in { options = { services.systemd-failmsg.enable = mkEnableOption "systemd-failmsg"; }; config = mkIf cfg.enable { environment.systemPackages = [ mypkgs.systemd-failmsg ]; systemd.packages = [ mypkgs.systemd-failmsg ]; }; } </code></pre> And enabling the service instead involved adding the following somewhere in my system's configuration: services.systemd-failmsg.enable = true; </code></pre> Conveniently, some Nextcloud services were failing on boot, which gave me a chance to test the failure message easily. Despite being able to execute the script manually successfully, when executed by systemd it couldn't find the failmsg.sh</code> script. I found the substituteInPlace</code> command in the manual and applied a quick fix to the derivation: postBuild = '' substituteInPlace failmsg@.service \ --replace /usr/bin/failmsg.sh $out/bin/failmsg.sh ''; </code></pre> The systemd service still failed, this time because it couldn't find the commands used, including hostname</code>, id</code>, sendmail</code>. A bit of digging around issues and the manual showed that I needed to set up the path correctly - makes sense, considering all the magic Nix does with $PATH</code>. I was initially considering using the substitueInPlace</code> command again to replace them with the appropriate executable path, e.g. --replace hostname ${coreutils.id}/bin/hostname</code>, but that didn't feel right. Looking at some more example and the manual, I found the writeShellApplication</code> builder. This is more meant for packages wherein the only output is a single shell script, but it automagically handles the script's $PATH</code> for you, so the script can use the commands like normal instead of absolute paths to the Nix store. In the end I saw wrapProgram</code> in some examples, which as the name suggests, wraps a program, setting environment variables as desired. I applied this to the failmsg.sh</code> script in the derivation, adding the required dependencies to $PATH</code>: let inputs = [ inetutils system-sendmail coreutils ]; in # ... nativeBuildInputs = [ makeWrapper ]; buildInputs = [ bash ] ++ inputs; installPhase = '' PREFIX=$out bash install.sh wrapProgram $out/bin/failmsg.sh \ --prefix PATH : ${lib.makeBinPath inputs} ''; </code></pre> (Question for the future: what differentiates nativeBuildInputs</code>, buildInputs</code>, and the like. Which are build-time only, which are runtime? What does "native" mean?) This resolved the issue with unavailable commands, but the service was still failing due to network unavailability while the service was starting up. The problem lies upstream, due to the service only containing After=network.target</code>, so the service doesn't have a hard requirement on network availability. I applied a fairly simple fix by adding a Requires</code> and After</code> to the systemd unit in the Nix module: systemd.services."failmsg@".unitConfig = { After = "network-online.target"; Requires = "network-online.target"; }; </code></pre> (TODO: should I submit this patch upstream?) Finally, the service is working as expected, and I'm content with its Nix implementation. Passing nix flake check</code></h2> I came across the nix flake check</code> command, which runs tests on your flake. So I thought it would be a good idea to ensure my flake passes. $ nix flake check error: Package ‘systemd-failmsg-1.3’ in /nix/store/...-source/nix/pkgs/systemd-failmsg/default.nix:22 is not supported on ‘x86_64-darwin’, refusing to evaluate. </code></pre> The first problem I hit was making my systemd-failmsg</code> package only available on Linux, otherwise the check produces the following error: It was surprisingly difficult to find the appropriate method for "only create this attribute if this condition holds". After a while I eventually remembered coming across the pattern of extending with an attribute set which may be empty depending on the condition. A simple enough function which is probably just implemented like so: c: set: if c then set else {} </code></pre> It took a surprising amount of time to come across the appropriate method. In the end I came across the pattern in the nixpkgs manual, using the function lib.attrsets.optionalAttrs</code>. Underneath it was the handy type specification in the lovely format I know from Haskell: optionalAttrs :: Bool -> AttrSet </code></pre> After discovering the NixOS options search</a> I can't help but wish there were a Hoogle</a> equivalent for Nix/nixpkgs library functions. Back to the problem at hand, I try using the condition pkgs.stdenv.isLinux</code>, but this causes a strange error on nix flake check</code>: error: attribute 'busybox' missing </code></pre> Adding --add-trace</code>, I eventually realised that at the very start of the trace, the system</code> causing the error is actually mipsel-linux</code>: … while checking the derivation 'packages.mipsel-linux.systemd-failmsg' at /nix/store/cl9c0n6wlgga14mcqkv7hypc6djdab00-source/nix/pkgs/default.nix:4:3: 3| // pkgs.lib.optionalAttrs pkgs.stdenv.isLinux { 4| systemd-failmsg = pkgs.callPackage ./systemd-failmsg.nix { }; | ^ 5| } … while checking flake output 'packages' at /nix/store/3hnh67ga9pw39j495b94h4fwhgqscm37-source/nix/flake.nix:13:7: 12| in rec { 13| packages = forAllSystems (system: import ./pkgs { | ^ 14| pkgs = import nixpkgs { inherit system; }; </code></pre> This system appears to currently be broken, so I'll try specifically filtering out mipsel-linux</code>. { pkgs }: { } // pkgs.lib.optionalAttrs (pkgs.stdenv.isLinux && pkgs.system != "mipsel-linux") { systemd-failmsg = pkgs.callPackage ./systemd-failmsg { }; } </code></pre> And nix flake check</code> passes! After coming across a common flake.nix</code> pattern, I decided to invert this system filtering, and instead only support a subset of platforms in my flake: { outputs = { self, nixpkgs, sops-nix }@inputs: let # Specify the list of supported platforms systems = [ "x86_64-linux" ]; forAllSystems = f: nixpkgs.lib.genAttrs systems (system: f system); in rec { packages = forAllSystems (system: import ./pkgs { pkgs = import nixpkgs { inherit system; }; }); }; } </code></pre> Considering only the platforms I care about seems like an easier way to go about this, rather than trying to support all the platforms Nix does. If I want to deploy to other platforms in the future I can simply add to systems</code>. Overlays: fixing nextcloud-news-updater</h2> Nextcloud's news-updater</a> is a handy tool for speedily updating RSS feeds. It unfortunately hard-codes running the occ</code> command under the Nextcloud installation directory. However, NixOS has its own nextcloud-occ</code> script which conveniently wraps the normal command with sudo</code> and the normal Nix stuff. I generally followed examples on Flake wiki page and managed to get it patched via an overlay. Worth noting I spent a long time trying to debug my first attempt, as nix flake check</code> would result in "error: infinite recursion encountered" in fixed-point.nix. It took me a while to find the issue was specifying the arguments as a set { self, super }:</code> instead of two separate arguments self: super:</code>. And this is why types and type checking is useful. That, and useful errors. Packaging Firefox Syncserver</a></h2> This was easily the most painful part of the conversion process. At the time of writing, there's no official package for this Python app in nixpkgs, because it's stuck on Python 2 and its dependencies have been broken. The new implementation syncstorage-rs</a> was recently packaged</a>, however,I wanted to stick with the old version I was already using. Also, the new app only supports MySQL and I didn't want to run yet another database engine on top of PostgreSQL, when a SQLite database would do just fine. Unfortunately there's no "official" tool for easily creating a Nix derivation for a Python packages, with all its dependencies nicely bundled up. I took a look at a few tools to do just that, with mixed success. Ideally what I want is a tool that I provide a package name (and maybe version), which then pulls down all its dependencies and generates Nix derivations for them as necessary, effectively pinning them in the Nix fashion. I wouldn't mind if this it autogenerated code for me, I just don't want to write it all myself. pypi2nix</a></h3> Searching through older Nix guides and discussion, pypi2nix was brought up as an easy way to import existing Python packages. However, the project's repository now is marked as archived, and it was indeed marked as broken in nixpkgs. I gave up on it quite quickly, looking for other suggested alternatives. mach-nix</a></h3> From the surface, it seems like a sensible system: you supply a list of requirements, just as in your average Python project, and dependency resolution is managed for you. However, I quickly ran into trouble with Python 2 support</a>, with many packages showing "not supported for interpreter python2.7" errors. mach-nix isn't able to handle evaluation errors due to packages marked as broken, which appears to be more of a problem with the Nix language</a>. This apparently makes it hard for mach-nix to do its dependency resolution without running into evaluation errors for Python 2 packages. I tried manually editing mach-nix to use tryEval</code> in a few more places, but I couldn't reliably fix the problem. Local testing in the nix repl</code> seemed to reliably prevent errors when evaluating derivations directly, but it failed to catch evaluation errors for a derivation's dependencies. It also wasn't easy to get a list of all of a derivations dependencies (see an attempt here</a>), which made trying to fix this even more tedious. At this point I also realised the underlying method mach-nix uses to make things "pure" as Nix requires: fetching a Git repository which indexes everything available in PyPi and other Python repositories. This index currently sits at over 400MB, which I'm sure will only grow with time. Personally, I felt this is a sledgehammer approach. I appreciate Nix already isn't particularly conservative when it comes to disk usage, but this is extreme. poetry2nix</a></h3> I initially thought poetry2nix was only for packages already using Poetry</a>, so I overlooked it in favour of trying out other Python-to-Nix tools. Reading a bit more into it, I realised I could repackage syncserver myself. I add python3Packages.poetry</code> to my dev shell, and go through the poetry init</code> process, requiring python 2.7</code>. But when trying to poetry add</code> some dependencies, it complains about the current Python version not being 2.7</code>. It's a bit confusing that a build tool needs to run on the version the end application uses, but I suppose it may need to evaluate the setup.py</code> script. This initially wasn't easy, poetry2nix doesn't officially support Python 2</a> because poetry</code> is marked as broken for Python 2 in nixpkgs, due to After running poetry env use python2.7</code>, it complains about my current version 2.7.xx</code> not matching 2.7</code>. Of course, I need to make the version requirement fuzzy by instead specifying ^2.7</code>. After the adjustment, poetry env use python2.7</code> works. poetry add</code> got a bit further now, but started complaining about the explicit version requirements conflicting with transitive dependencies. To my horror, it turns out pip</code> doesn't do proper dependency resolution, apparently it simply accepts the first version of a package that is requested. Naturally, Poetry doesn't provide an escape-hatch for this situation</a>, making it a pain to port packages relying on this broken pip</code> behaviour to Poetry. My solution was to provide to include and patch a nested copy of the broken dependency. This could probably be replaced with the usual Nix source patching mechanism along with a poetry2nix override. Before working out how poetry2nix overrides worked, I tried manually adjusting the poetry.lock</code> file to change the source of another broken dependency, umemcache</a>, which has unreleased fixes. Annoyingly, I hit a poetry bug which interfered with poetry2nix</a>, where dependency metadata wasn't being specified, which is what poetry2nix relies on to make all the dependency fetching pure. I fixed this with a nix flake update</code> to nixpkgs. Unfortunately this introduced more issues, including an infinite recursion error</a>, and a large number of packages requiring setuptools</code> to be added to their dependencies. I managed to work around both with a tedious number of poetry2nix overrides. Another limitation I hit was Poetry not locking URL dependencies</a>, causing confusing errors about hashes being required when building with poetry2nix. The fix is to override the src</code> of the offending packages, repeating the URL and providing the correct hash. After this, I finally had a build-able package, I then used the dependencyEnv</code> attribute to get a store path containing all the dependencies, including Gunicorn which I needed my service to execute. I hit a collision between some files, so I had to set ignoreCollisions = true;</code>: Collision between backports-functools-lru-cache and python2.7-configparser: python2.7/site-packages/backports/init.pyc </code></pre> Configuring the service</h3> From there, I amended my NixOS module for firefox-syncserver until it was up and running. Some fun problems: Nix: confusing errors when generating the config file with format.generate</code> and recursiveUpdate</code>. I've forgotten the details of what caused this. Gunicorn: fun trial and error with the systemd service hardening, additionally needed the @chown</code> and @setuid</code> permissions, for changing the owner of a file in /tmp</code>, and setting the user for worker processes respectively. Gunicorn: having no logging by default! And making it tedious to configure logging. Syncserver: forgetting the https://</code> in the public_url</code> config option causes the annoying mismatch between public_url</code> and the origin URL received by Gunicorn. The error message also unhelpfully duplicates the received URL, causing me to confuse the issue for a bug in Nginx or Gunicorn itself. My website</h2> I use a fairly simple deployment method for my website: $ git push</code> with a post-receive hook</a> on the receiving end, which rebuilds the website using Zola</a> (and Graphviz</a> for some diagrams). This introduces a problem to the Nix system configuration for the webserver: cloning a Git repository and updating it separately is an impure operation. It doesn't make much sense to wrap the website in a Nix package if I want to be able to painlessly update the website separately to the system it runs on. However, I would like the NixOS configuration to include the initial setup of the repository (cloning it, and setting up that post-receive hook). While searching for potential methods of implement this, I came across system.activationScripts</code></a> in other people's configurations. This seems to fit the bill - it allows you to specify arbitrary scripts that run on boot and nixos-rebuild</code>. I created a script that clones the repository to an appropriate location, and installs a wrapped version of the post-receive hook. Deploying</h1> I used nixos-infect</a>, as a cloud-init script when allocating a VM on Hetzner (with Ubuntu as the base OS). It works perfectly, and after a reboot the new VM claims it is NixOS: $ cat /etc/os-release BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues" BUILD_ID="22.05.3475.ed9b904c5eb" DOCUMENTATION_URL="https://nixos.org/learn.html" HOME_URL="https://nixos.org/" ID=nixos LOGO="nix-snowflake" NAME=NixOS PRETTY_NAME="NixOS 22.05 (Quokka)" SUPPORT_URL="https://nixos.org/community.html" VERSION="22.05 (Quokka)" VERSION_CODENAME=quokka VERSION_ID="22.05" </code></pre> Then I attempted to deploy my system configuration with nixos-rebuild</code>: $ nixos-rebuild nixos-rebuild switch --target-host atlas2 --flake .#webserver error: Failed assertions: - The ‘fileSystems’ option does not specify your root file system. - You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable. </code></pre> Probably missing the /etc/nixos/hardware-configuration.nix</code>. Copied it from the VM and included it in the system config. Another error: error: Failed assertions: - You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable. </code></pre> Weird that nixos-infect doesn't sort out the bootloader for you. I manually set to boot.loader.grub.device = '/dev/sda';</code> and that sorted that. The next deploy worked, but there were a couple of errors, because I forgot to update the sops keys. I also messed up and set PermitRootLogin no</code> in the SSH config without setting up a user with sudo</code> access, so I rolled back over the existing SSH connection: $ nixos-rebuild switch --rollback </code></pre> Glad to see that switching works without rebooting! It even disables & stops services, deletes groups, and all that stuff. While sorting this out, it was annoying having to swap my local SSH config back-and-forth between root and a normal user. Found the --use-remote-sudo</code> option for nixos-rebuild</code>, but this seems to require passwordless sudo. I set that up with the following options: users.users.<user>.extraGroups = [ "wheel" ]; security.sudo.wheelNeedsPassword = false; </code></pre> Re-did the deploy with --use-remote-sudo</code> and it had the same result, so I'm taking it as done. Some strange errors on the next rebuild: error: cannot add path '/nix/store/1rhrxs9n736a4f3gqlqmi211xnhi10ka-nextcloud-config.php' because it lacks a valid signature </code></pre> From this issue</a>, it appears to be caused by nixos-rebuild</code> copying things from my local store to the server's store - by default copying is apparently only allowed from cache.nixos.org</code>. At this point I messed up the SSH and sudo permissions and was locked out of root. I also didn't have a root password, so I went through Hetzner rescue. The lack of a normal $PATH</code> made it somewhat tedious to run the commands needed, including bash</code> and passwd</code>. I had to search the store for them. The "valid signature" issue turned out to be due to swapping to the normal user with --use-remote-sudo</code>. Apparently one needs to tweak nix.settings.trusted-users</code></a> to include any users that are able to nixos-rebuild</code>. To resolve this, I added @wheel</code> to this setting, and ran nixos-rebuild</code> from the machine itself to work around the issue. I added this to the wiki page</a> to highlight the problem before it occurs. Performing the migration</h2> At this point I had a fully functional NixOS system, reproducing just about everything I needed from the old server! As a temporary measure, I replicated my normal website DNS entries but adding a prefix: new.averyv.me</code> - this was quite easy since I'd made the domain name a Nix option. I followed some standard data migration procedures, copying over my Nextcloud data and PostgreSQL database. Things generally "just worked", though I noticed a few odd things that needed minor adjustments to options. Some testing ensured, I made sure all the new services worked by logging into them on my devices. After some more hassle with firefox-syncserver, I was fairly confident all was in order, so I migrated across by editing the normal DNS entries to point to the new server's IP address. Book review: Meditations, Marcus Aurelius Antoninus Unknown — Thu, 13 Jan 2022 00:00:00 +0000 Introduction</h2> Having recently decided to do some more (non-fictional) reading, I started by picking up some literature already available to me, starting with Meditations. My copy was translated and annotated by Martin Hammond, and published by Penguin Classics.1</a> The Meditations are the surviving writings of Marcus Aurelius, created later in his life and not intended to be published. Nevertheless, it serves as an interesting glimpse into the mind of a particularly interesting and (for his time) important character. Most entries ("chapters") serve as reminders to himself for how he should think or behave, or justify his beliefs with rhetorical arguments. Hammond's notes provide some helpful context, and regularly spares you from jumping to your own research when Marcus brings up things you're unfamiliar with - though some have been lost to history. He also links related chapters together, which is handy for jumping through Marcus' repeated explorations of the same idea. Summary</h2> I'll summarise the ideas and themes that stood out in my reading of the Meditations. A universal order</h3> Marcus regularly talks about a universal order which he believes governs all things. He refers to this order in various ways, including god / the gods, Nature, and the Whole. He describes this order almost as if it were conscious: Nothing which benefits the Whole can be harmful to the part, and the Whole contains nothing which is not to its benefit. _{Extract from Book 10, Chapter 6 </blockquote>} From this perspective he insists on the existence of the gods, and that the order they bring about is inherently good: Now if the gods took thought for me and for what must happen to me, they will have taken thought for my good. It is not easy to conceive of a thoughtless god, and what possible reason could they have had to be bent on my harm? What advantage would there have been from that either for themselves or for the common good, which is the main concern of their providence? If they did not take individual thought for me, then certainly they took thought for the common good, and since what happens to me is a consequential part of that, I should accept and welcome it. _{Extract from Book 6, Chapter 44 </blockquote>} The divinity of rationality</h3> A quintessential idea of Stoicism is the importance of human logic and rationality, and Marcus applies a religious fervour to this. He describes the directing mind as the "god within". While he also maintains the existence of the actual gods (the "god without"), it is rather striking to deify rationality itself. He later applies this concept when discussing good and evil, combining it with the universal order: The sinner sins against himself: the wrongdoer wrongs himself, by making himself morally bad. _{Book 9, Chapter 4 </blockquote>} Humans sin against the universal order and themselves when they act irrationally, and when they do not accept - or contravene - things brought about by Nature: The soul of a man harms itself, first and foremost, when it becomes (as far as it can) a separate growth, a sort of tumour on the universe: because to resent anything that happens is to separate oneself in revolt from Nature, which holds in collective embrace the particular natures of all other things. Secondly, when it turns away from another human being, or is even carried so far in opposition as to intend him harm - such is the case in the souls of those gripped by anger. [...] Fourthly, whenever it dissimulates, doing or saying anything feigned or false. [...] And the end for rational creatures is to follow the reason and the rule of that most venerable archetype of a governing state - the Universe. _{Extract from Book 2, Chapter 16 </blockquote>} Marcus regularly argues to treat these sinners with patience, as they do not understand their own actions. This serves mostly as a reminder to himself, and he appears to have struggled in applying this throughout his life. When someone does you some wrong, you should consider immediately what judgement of good or evil led him to wrong you. When you see this, you will pity him, and not feel surprise or anger. You yourself either still share his view of good, or something like it, in which case you should understand and forgive: if, on the other hand, you no longer judge such things as either good or evil, it will be the easier for you to be patient _{Book 7, Chapter 26 </blockquote>} Purpose</h3> While Marcus is strongly theistic, and insists in the existence of the gods, he doesn't tie purpose to them: The Whole is either a god - then all is well: or if purposeless - some sort of random arrangement of atoms or molecules - you should not be without purpose yourself. _{Extract from Book 9, Chapter 28 </blockquote>} Humans are created for each other, and all inhabit the common City of Zeus. Their purpose is thus to serve each other, and to contribute to the 'common good' of all mankind: Best for each is what suits his own condition and nature: and my nature is both rational and social. As Antoninus, my city and country is Rome: as a human being, it is the world. So what benefits these two cities is my only good. _{Extract from Book 6, Chapter 44 </blockquote>} Every creature must do what follows from its own constitution. The rest of creation is constituted to serve rational beings (just as in everything else the lower exists for the higher), but rational beings are here to serve each other. So the main principle in man's constitution is the social. _{Extract from Book 7, Chapter 55 </blockquote>} As mankind's purpose is a social one, to separate oneself from society is contrary to Nature. The same is true for actively harming each other through words or actions. Accepting one's own lot</h3> Given Marcus' establishment of a benevolent universal order, he concludes that each person's position is as it should be, so one that accepts what life gives them is in accordance with Nature: All that you pray to reach at some point in the circuit of your life can be yours now - if you are generous to yourself. That is, if you leave all the past behind, entrust the future to Providence, and direct the present solely to reverence and justice. To reverence, so that you come to love your given lot: it was Nature that brought it to you and you to it. _{Extract from Book 12, Chapter 1 </blockquote>} Antoninus does not entirely ignore that his own lot in life differs significantly from that of others: How clearly it strikes you that there is no other walk of life so conducive to the exercise of philosophy as this in which you now find yourself! _{Book 11, Chapter 7 </blockquote>} He does also believe that change to the status quo is required, particularly in that which contradicts the natural order: Farce, war, frenzy, torpor, slavery! Day by day those sacred doctrines of yours will be wiped out, whenever you conceive and admit them untested by natural philosophy. _{Book 10, Chapter 9 </blockquote>} Don't hope for Plato's Utopian republic, but be content with the smallest step forward, and regard even that result as no mean achievement. _{Extract from Book 9, Chapter 29 </blockquote>} Perhaps his position is that one should accept the lot Nature gives you, but should work against the unnatural acts by irrational people contravening the Whole? Emotional control</h3> As one's emotions inhibits one's rationality, they should be controlled to not tarnish the divinity of the directing mind. Marcus describes some sort of meditation as a method of control over one's mind and body: Finally, then, remember this retreat into your own little territory within yourself. Above all, no agonies, no tensions. Be your own master, and look at things as a man, as a human being, as a citizen, as a mortal creature. And here are two of the most immediately useful thoughts you will dip into. First that things cannot touch the mind: they are external and inert; anxieties can only come from your internal judgement. Second, that all these things you see will change almost as you look at them, and then will be no more. _{Extract from Book 4, Chapter 3.4 </blockquote>} He also encourages mastery over both pleasure and pain: He responds to the divinity seated within him, and this renders the man unsullied by pleasures, unscathed by any pain, untouched by any wrong, unconscious of any wickedness; a wrestler for the greatest prize of all, to avoid being thrown by any passion [...] _{Extract from Book 3, Chapter 4.3 </blockquote>} Pain is something to be borne... Whenever you suffer pain, have ready to hand the thought that pain is not a moral evil and does not harm your governing intelligence: pain can do no damage either to its rational or to its social nature. _{Extract from Book 7, Chapter 64 </blockquote>} While pleasure can draw us in and control us, preventing us from serving the Whole: Because it is not right that the rational and social good should be rivalled by anything of a different order, for example the praise of the many, or power, or wealth, or the enjoyment of pleasure. All these things may seem to suit for a little while, but they can suddenly take control and carry you away. _{Extract from Book 3, Chapter 6.3 </blockquote>} Marcus applies an analytical reductionism to things in order to control his reactions to them. Things that cause pleasure and pain are superficial, and breaking them down provides a sober perspective - once these shallow things are cast off, one can instead focus on the important. Ask then, what is this which is now making its impression on me? What is it composed of? How long in the nature of things will it last? What virtue is needed to meet it - gentleness, for example, or courage, truthfulness, loyalty, simplicity, self- sufficiency, and so on? _{Extract from Book 3, Chapter 11.2 </blockquote>} And he insists that 'all is as thinking makes it so' - the mind's reaction produces these emotions, positive and negative. Control over one's thoughts provides control over one's feelings. Look at causation stripped bare of its covers; look at the ulterior reference of any action. Consider, what is pain? What is pleasure? What is death? What is fame? Who is not himself the cause of his own unrest? Reflect how no one is hampered by any other; and that all is as thinking makes it so. _{Book 12, Chapter 8 </blockquote>} Life and death</h3> Two recurring themes are the ever-changing nature of the universe, and the idea that same sorts of events occur repeatedly in cycles. Marcus reminds us of the short amount of time we have, highlighting that we should endeavour to use it to the best of our abilities - bettering ourselves and serving the Whole. It is high time now for you to understand the universe of which you are a part, and the governor of that universe of whom you constitute an emanation: and that there is a limit circumscribed to your time - if you do not use it to clear away your clouds, it will be gone, and you will be gone, and the opportunity will not return. _{Extract from Book 2, Chapter 4 </blockquote>} On the other hand, the sameness of things appears to provide a comfort when considering death, as past a certain point there will be nothing new. Death is viewed as part of the natural order of things, and is therefore something to be accepted rather than feared: 'it is nothing more than a function of nature - and if anyone is frightened of a function of nature, he is a mere child' (Book 2, Chapter 12). It is also viewed as something of a great equaliser - as all share the same fate. Simultaneously, the ever-changing universe will quickly wash away most if not all evidence of your life, therefore to be concerned with fame and legacy is a foolish waste of time. So always remember these two things. First, that all things have been of the same kind from everlasting, coming round and round again, and it makes no difference whether one will see the same things for a hundred years, or two hundred years, or for an infinity of time. Second, that both the longest-lived and the earliest to die suffer the same loss. It is only the present moment of which either stands to be deprived: and if indeed this is all he has, he cannot lose what he does not have. _{Book 2, Chapter 14.2 </blockquote>} Personal thoughts</h2> Marcus' misanthropy</h3> It is quite clear that Marcus held some contempt for his peers. Interspersed with his regular reminders that 'all men are brothers' and that sinners act out of ignorance are vague entries in which he criticises unnamed others: A black character, an effeminate, unbending character, the character of a brute or dumb animal: infantile, stupid, fraudulent, coarse, mercenary, despotic. _{Book 4, Chapter 28 </blockquote>} Children's tantrums and toys, 'tiny spirits carrying corpses' - the Underworld in the Odyssey strikes more real! _{Book 9, Chapter 24 </blockquote>} The conflicting nature of these entries highlights the difficulties Marcus had in practicing his own philosophy - of to loving and living for mankind. The regularity in which such passages appear in the Meditations suggest this remained a problem throughout his life. He recommends patience in dealing with 'sinners' and other flawed individuals, but also reminds himself that sometimes his judgment maybe incorrect: If he did wrong, the harm is to himself. But perhaps he did not do wrong. _{Book 9, Chapter 24 </blockquote>} It's rather interesting to see the combination of self-criticism and criticism of others. Marcus appears was fully cognizant of his own failures in adhering to his philosophical beliefs, showing some humility in the man who held one of the most powerful positions of his time. Marcus' obsessive reductionism</h3> While I can respect the criticism of giving in to emotion and allowing it to affect your rationality, Marcus occasionally takes this to the extreme. You will think little of the entertainment of song or dance or all-in wrestling if you deconstruct the melodic line of a song into its individual notes and ask yourself of each of them: 'Is this something that overpowers me?' You will recoil from that admission. So too with a comparable analysis of dance by each movement and each pose, and the same again with wrestling. Generally, then, with the exception of virtue and its workings, remember to go straight to the component parts of anything, and through that analysis come to despise the thing itself. And the same method should be applied to the whole of life. _{Book 9, Chapter 24 </blockquote>} Here he applies his deconstruction to music and similar arts. This absolutism strikes me as unhealthy - a balance can be struck between allowing yourself enjoyment and indulging in them excessively. I expect most people would struggle to maintain such an ideal, and have to wonder how it affected Marcus - whether these words were a symptom of his misanthropy or a depression. Similar deconstruction of sexual matters, combined with his thankfulness at having left sex to his later years, marks him as a rather asexual individual. This interpretation could be entirely misjudged, considering he had 14 children. He doesn't go into much detail when discussing his own sexual activities, but we can assume he'd be strongly in favour of controlling one's lust. An emperor accepting his own lot</h3> Marcus' arguments for 'accepting one's own lot' didn't sit right with me when considering that they were written by an emperor. Certainly his position gave him liberties and comforts few others possessed, so it is easy to scoff at an emperor telling himself to accept what life has given him. This emperor, however, seems to regard his royal status with disdain quite often - 'Let nobody any more hear you blaming palace life: don't hear yourself blaming it.' (Book 8, Chapter 9). He comments on the conflicts between his ideals of virtue and the behaviour commonly attributed to those who have shared his station. On the other hand, he also sees that he can exercise his virtues particularly effectively in his walk of life (see earlier quotation of Book 11, Chapter 7). Marcus does give the impression of a man striving to be a good king according to his own ideals, and shows some vision for a better future. Though he spent most of his rule in military campaigns, he apparently spent some of his time dealing with legal matters involving freeing slaves and managing guardianship, and respected the Senate more than other Roman dictators before and after him. Perhaps this philosophy is just his version of Divine Right, with which he justifies his position - Roman emperors were deified after all. Or perhaps that is being too harsh, and as Marcus quotes: 'A king's lot: to do good and be damned'. Accepting one's lot in life does appear to be a common Stoic teaching, so it's not entirely fair to complain of Marcus applying it to his own life. Emotions are natural but must be controlled</h3> For someone who worships Nature, arguing for mankind to exert control over their emotions strikes me as a strange contradiction. Surely pleasure and pain, which are natural and instinctual parts of us, are not inherently bad? The conflict here appears to be between nature and rationality, both things which Marcus worships in a way. Does the divinity of rationality supersede the divinity of Nature? This seems to be an implicit conclusion in his writing. Perhaps the justification is that emotions are more animalistic than rationality. Despite both having come about naturally, logic developed later as we evolved intelligence, so the latter should stay in control. Conclusion</h2> The Meditations are a curiosity, but I must admit I didn't find it the most pleasant or smoothest of journeys. It took significantly longer to read through compared to my average book, and I had to digest it small pieces at a time. This may be partly due to the dryness of the subjects - no, I don't read much philosophy - but also the nature of a personal journal. Antoninus used his writings to convince himself of things, so ideas, arguments, and beliefs are repeatedly mentioned and reformulated in different ways. I recommend jumping between the translated text and notes to get additional context while the chapters are still fresh your mind. The text doesn't serve as the best introduction into Stoicism - it being the opinions of just one man who is considered to be a member of the school of thought. A better overview might be gained from a combination of multiple perspectives, exploring how Stoicism is applied in practice, how it has developed over time. On top of that, the religious flavour Marcus applies in his philosophy certainly doesn't suit me. If you're particularly enamoured with ancient Roman history, you may find it interesting to delve into the minds of one of its significant figures. However, if you're interested in the emperor recounting and discussing contemporary events, you'll be left disappointed. Some of Marcus Aurelius' ideas are timeless, but you can probably explore them better elsewhere. ^{1 Meditations, Marcus Aurelius, Diskin Clay (Introducer), Martin Hammond (Translator), Penguin Classics, 2006, ISBN 978-0140449334 </div>} Splitting up Liquibase JSON changelogs Unknown — Thu, 09 Sep 2021 00:00:00 +0000 When migrating to Liquibase</a> for your database deployments, a common practice is to create a starting point from an existing schema in a database using the generate-changelog</code></a> command. This command generates a Liquibase changelog which will recreate your schema when you run $ liquibase update</code>. Unfortunately, at least in the open source version</a>, this command generates a single file containing all the changes. I haven't tried the "Pro" version of Liquibase, but it can apparently split up the changelog by creating auxiliary files for the different types of database object. If your existing schema already has a lot of tables, the generated changelog can be huge, even just for the tables, indexes, and foreign keys. This can be quite an organisational nightmare, and is difficult to read through manually. But if you do want to split up your changelog, and you use the JSON changelog format</a>, some jq</code></a> magic might just do the trick. Combine it with some Bash and you can fully automate the splitting process that would be otherwise incredibly tedious. Here's a jq</code> program that splits up a single changelog into smaller changelogs that all relate to a single table - including the createTable</code>, createIndex</code>, and addForeignKeyConstraint</code> changes: # Define a function for mapping a changes onto its destination file name # createTable and createIndex use the tableName field # addForeignKeyConstraint uses baseTableName # Default to using the name of the change, e.g. createSequence def get_change_group: map(.tableName // .baseTableName)[0] // keys[0]; # Select the main changelog object .databaseChangeLog # Collect the changes from each changeSet into an array | map(.changeSet.changes | .[]) # Group changes according to the grouping function | group_by(get_change_group) # Select the grouped objects from the array | .[] # Get the group name from each group | (.[0] | get_change_group) as $group_name # Select both the group name... | $group_name, # and the group, wrapped in a changeSet that uses the group name in the ID and # the current user as the author { databaseChangelog: { changeSet: { id: ("table_" + $group_name), author: env.USER, changes: . } } } </code></pre> Save this program as split_liquibase_changelog.jq</code>. Using, e.g., a Bash script, you can use this to split changelogs into smaller files. But first, we need another program to generate a main changelog file. This one will simply include all of the generated changelogs: Finally, here's a Bash script that ties it all together: #!/usr/bin/env bash # Example: ./split_liquibase_changelog.sh schema < changelog.json set -e -o noclobber OUTPUT_DIRECTORY="${1:-schema}" OUTPUT_FILE="${2:-schema.json}" # Create the output directory mkdir --parents "$OUTPUT_DIRECTORY" # --raw-output: don't quote the strings for the group names # --compact-output: output one JSON object per line jq \ --raw-output \ --compact-output \ --from-file split_liquibase_changelog.jq \ | while read -r group; do # Read the group name line # Read the JSON object line read -r json # Process with jq again to pretty-print the object, then redirect it to the # new file (jq '.' <<< "$json") \ > "$OUTPUT_DIRECTORY"/"$group".json done # List all the files in the input directory # Run jq with --raw-input, so input is parsed as strings # Create a changelog that includes everything in the input path # Save the output to the desired output file (jq \ --raw-input \ '{ databaseChangeLog: [ { includeAll: { path: . } } ] }' \ <<< "$OUTPUT_DIRECTORY"/) \ > "$OUTPUT_FILE" </code></pre> Alternatively, if you'd prefer explicitly including each smaller changeset, instead use the following at the end of the script: # List all the files in the input directory # Run jq with: # --null-input: don't read the input, so `inputs` refers to ALL the inputs # --raw-input: parse input as strings # Save the output to the desired output file find "$OUTPUT_DIRECTORY" -type f \ | jq \ --null-input \ --raw-input \ ' # Wrap all the inputs (file names) into an array [inputs] # Create the main changelog object with an include for each file | { databaseChangeLog: map({ include: { file: . } }) } ' \ > "$OUTPUT_FILE" </code></pre> Now you can run $ ./split_liquibase_changelog.sh schema schema.json < changelog.json</code> and enjoy your beautifully organised changelogs! Make sure to check everything is correct and organised as you want it. Feel free to adjust the programs if you want to tweak how things are organised, you may want to adjust: The files that don't relate to any table - these will be named by the command, e.g. createSequence</code>.</li> The IDs of the changesets in these files, which will be nonsensically prefixed with table_</code>.</li> The grouping function get_change_group</code> in generate_liquibase_includes</code></li> How the smaller changelog files group all the changes into a single changeset</li> </ul> Take a look at jqplay</a> if you want to fiddle with jq</code> programs. Example</h1> Here's a contrived example based on my Nextcloud instance's database. Generated changelog</summary> { "databaseChangeLog": [ { "changeSet": { "id": "1631138109568-75", "author": "avery (generated)", "changes": [ { "createTable": { "columns": [ { "column": { "autoIncrement": true, "constraints": { "nullable": false, "primaryKey": true, "primaryKeyName": "oc_news_feeds_pkey" }, "name": "id", "type": "BIGINT" } }, { "column": { "constraints": { "nullable": false }, "defaultValueNumeric": 0, "name": "update_mode", "type": "INTEGER" } }] , "tableName": "oc_news_feeds" } }] } }, { "changeSet": { "id": "1631138109568-240", "author": "avery (generated)", "changes": [ { "addForeignKeyConstraint": { "baseColumnNames": "feed_id", "baseTableName": "oc_news_items", "constraintName": "feed", "deferrable": false, "initiallyDeferred": false, "onDelete": "CASCADE", "onUpdate": "NO ACTION", "referencedColumnNames": "id", "referencedTableName": "oc_news_feeds", "validate": true } }] } }, { "changeSet": { "id": "1631138109568-236", "author": "avery (generated)", "changes": [ { "createIndex": { "columns": [ { "column": { "defaultValueNumeric": 0, "name": "last_modified" } }] , "indexName": "news_feeds_last_mod_idx", "tableName": "oc_news_feeds" } }] } }, { "changeSet": { "id": "1631138109568-241", "author": "avery (generated)", "changes": [ { "createIndex": { "columns": [ { "column": { "defaultValueNumeric": 0, "name": "last_modified" } }] , "indexName": "news_items_last_mod_idx", "tableName": "oc_news_items" } }] } } ]} </code></pre> </details> New files created after running split_liquibase_changelog.sh</code>: schema/oc_news_feeds.json</summary> { "databaseChangelog": { "changeSet": { "id": "table_oc_news_feeds", "author": "avery", "changes": [ { "createTable": { "columns": [ { "column": { "autoIncrement": true, "constraints": { "nullable": false, "primaryKey": true, "primaryKeyName": "oc_news_feeds_pkey" }, "name": "id", "type": "BIGINT" } }, { "column": { "constraints": { "nullable": false }, "defaultValueNumeric": 0, "name": "update_mode", "type": "INTEGER" } } ], "tableName": "oc_news_feeds" } }, { "createIndex": { "columns": [ { "column": { "defaultValueNumeric": 0, "name": "last_modified" } } ], "indexName": "news_feeds_last_mod_idx", "tableName": "oc_news_feeds" } } ] } } } </code></pre> </details> schema/oc_news_items.json</summary> { "databaseChangelog": { "changeSet": { "id": "table_oc_news_items", "author": "avery", "changes": [ { "addForeignKeyConstraint": { "baseColumnNames": "feed_id", "baseTableName": "oc_news_items", "constraintName": "feed", "deferrable": false, "initiallyDeferred": false, "onDelete": "CASCADE", "onUpdate": "NO ACTION", "referencedColumnNames": "id", "referencedTableName": "oc_news_feeds", "validate": true } }, { "createIndex": { "columns": [ { "column": { "defaultValueNumeric": 0, "name": "last_modified" } } ], "indexName": "news_items_last_mod_idx", "tableName": "oc_news_items" } } ] } } } </code></pre> </details> schema.json</summary> { "databaseChangeLog": [ { "include": { "file": "schema/oc_news_items.json" } }, { "include": { "file": "schema/oc_news_feeds.json" } } ] } </code></pre> </details> Git from first principles Unknown — Tue, 06 Jul 2021 00:00:00 +0000 Introduction</h2> Git</a> is the best version control software I've used, but it has a reputation for being complex and confusing, in part thanks to confusing commands and lackluster documentation. In this article I've tried to create my own guide that'll provide readers with an understanding - and hopefully confidence - when using Git. Even if you've been a Git user for a while, you might learn a thing or two - I certainly did while I was writing this. I think learning some of Git's internal concepts and mechanisms is crucial for understanding how to use the tool as a whole. These details are normally hidden by the user-friendly Git commands and graphical Git clients. Without knowing what Git commands are doing, it's easy to get lost when something goes wrong and not know how to recover. Instead of hiding them, I'll explain the some of the under-the-hood behaviour while teaching the day-to-day Git commands. If you prefer video explanations over long text articles like this, I'd recommend checking out The Missing Semester's lecture on Git,1</a> Git For Ages 4 And Up for Git beginners,2</a> or Git From the Bits Up for those who are more experienced.3</a> This article takes inspiration from these, but also tries to expand on what they didn't have time to cover. Getting started</h2> Let's start by checking the standard Git help: $ git help usage: git [--version] [--help] [-C <path>] [-c <name>=<value>] [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path] [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare] [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>] <command> [<args>] These are common Git commands used in various situations: start a working area (see also: git help tutorial) clone Clone a repository into a new directory init Create an empty Git repository or reinitialize an existing one work on the current change (see also: git help everyday) add Add file contents to the index mv Move or rename a file, a directory, or a symlink restore Restore working tree files rm Remove files from the working tree and from the index sparse-checkout Initialize and modify the sparse-checkout examine the history and state (see also: git help revisions) bisect Use binary search to find the commit that introduced a bug diff Show changes between commits, commit and working tree, etc grep Print lines matching a pattern log Show commit logs show Show various types of objects status Show the working tree status grow, mark and tweak your common history branch List, create, or delete branches commit Record changes to the repository merge Join two or more development histories together rebase Reapply commits on top of another base tip reset Reset current HEAD to the specified state switch Switch branches tag Create, list, delete or verify a tag object signed with GPG collaborate (see also: git help workflows) fetch Download objects and refs from another repository pull Fetch from and integrate with another repository or a local branch push Update remote refs along with associated objects 'git help -a' and 'git help -g' list available subcommands and some concept guides. See 'git help <command>' or 'git help <concept>' to read about a specific subcommand or concept. See 'git help git' for an overview of the system. </code></pre> All the commands listed above are called porcelain commands; they are high-level and user-friendly, and all you need when using Git day-to-day. Porcelain commands use plumbing to get stuff done under the hood. Plumbing are a separate set of fundamental data structures and utilities. These terms an analogy, comparing this dichotomy to the porcelain and plumbing found in a bathroom: the pretty porcelain hides away the confusing plumbing, which you usually don't want to think about. I'll explain Git-specific terms as we get to them, but you can refer to the Git glossary with $ man gitglossary</code>4</a> for an explanation of any new terms you come across. Creating a repository</h2> A Git repository is a directory with a specific structure and set of files, storing all of the data Git needs. To create a new Git repository, navigate to an empty directory and run $ git init</code>: $ git init $ tree -a . └── .git ├── branches ├── config ├── description ├── HEAD ├── hooks │ ├── applypatch-msg.sample │ ├── commit-msg.sample │ ├── fsmonitor-watchman.sample │ ├── post-update.sample │ ├── pre-applypatch.sample │ ├── pre-commit.sample │ ├── pre-merge-commit.sample │ ├── prepare-commit-msg.sample │ ├── pre-push.sample │ ├── pre-rebase.sample │ ├── pre-receive.sample │ ├── push-to-checkout.sample │ └── update.sample ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags </code></pre> Git has now created a new hidden directory called .git</code> - your new repository. Your previously empty directory is known as your working tree (a.k.a. worktree or working tree). It's a regular directory on your computer, but Git will control its contents. Running Git commands will store the contents of files from the working tree into the Git repository, as well as restore files stored in the repository to the working tree. When you run $ git ...</code>, Git searches your current directory and its parents for a valid repository in a .git</code> directory. I'll remove the description</code> file and hooks/</code> directory, since they're optional. You can see that an empty repository is pretty simple: .git ├── branches ├── config ├── HEAD ├── info │ └── exclude ├── objects │ ├── info │ └── pack └── refs ├── heads └── tags </code></pre> The config</code> file stores your repository-specific configuration - options set with $ git config</code> are stored in this file. To start off, set your username and email, which will be saved along with the changes you make: $ git config user.name Avery $ git config user.email avery@example.com # The [user] section has been added # The [core] section contains some internal Git configuration $ cat .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [user] name = Avery email = avery@example.com </code></pre> I'll explain the rest of the files and directories as we get to them. Saving snapshots with commits</h2> The purpose of version control is to store snapshots of a directory tree (in this case the files within your working tree). Git stores these snapshots with commits. Commits contain a snapshot of all the files, and some contextual information, such as the author and a message to explain what was changed from the previous snapshot. Let's step through the process and explain how Git creates a commit. First, let's check the current status of our working tree: $ git status On branch master No commits yet nothing to commit (create/copy files and use "git add" to track) </code></pre> This tells us that: We're on the default branch of the repository, called master</code></li> We haven't committed anything yet (i.e. Git hasn't saved any snapshots)</li> Our working tree has no changes that could be committed</li> </ol> I'll go into branches later. For now, let's create a small text file so our commit has something to take a snapshot of: $ echo "Some contents" > file1.txt $ git status On branch master No commits yet Untracked files: (use "git add <file>..." to include in what will be committed) file1.txt nothing added to commit but untracked files present (use "git add" to track) </code></pre> Git helpfully tells us that we need to add</code> file1.txt</code> for Git to track it (i.e. include it in snapshots) so let's do that: $ git add file1.txt $ git status On branch master No commits yet Changes to be committed: (use "git rm --cached <file>..." to unstage) new file: file1.txt </code></pre> Using $ git add</code> has added file1.txt</code> to the staging area. It tracks the contents that will be saved in the next snapshot, which can differ from the contents of your working tree. This is handy when you've made several unrelated changes in your working tree, and you want to save them as separate commits. Think of the staging area as a virtual working tree contained within the repository - running $ git add</code> copies the contents of files from your working tree into the staging area. When you create a commit, Git saves a copy of the staging area as the snapshot of the directory. The status now shows file1.txt</code> under "Changes to be committed", indicating that it is staged. You could also have changes in tracked files that aren't staged yet, in which case they would be listed under "Changes not staged for commit". When you add content to the staging area, Git immediately saves it - specifically, it saves a snapshot of each file in the staging area. After $ git add</code>ing file1.txt</code>, some new things are added to the repository: .git ├── branches ├── config ├── HEAD ├── index ├── info │ └── exclude ├── objects │ ├── 1e │ │ └── d6543483aafc93c5323daea1860bd7a29857d4 │ ├── info │ └── pack └── refs ├── heads └── tags </code></pre> We now have two new files: index</code>, and a file in objects</code> known as an object. There are three different types of Git objects: blobs, trees, and commits. The file in objects/1e/</code> is a blob: this type contains metadata and the (compressed) contents of a file. In this case it's the contents of file1.txt</code> when you $ git add</code>ed it. The full name of an object starts with the name of the subdirectory it's contained in. You can check the contents of the blob by using $ git show</code>. # Prepend the "1e" from the subdirectory $ git show 1ed6543483aafc93c5323daea1860bd7a29857d4 Some contents </code></pre> If you've been running these commands yourself, you might notice that the name of the object is different in your repository - that's likely because you didn't put exactly the same contents in your file1.txt</code>, or the metadata might be different if you're using a different version of Git. The object's name (and hence file path) is actually a hash of its contents. The hash represents the contents of the object with a unique fixed-length string. Due to the additional metadata in the object, it won't simply be the hash of the contents of file1.txt</code>. As of writing, the hash algorithm used is SHA-1, but Git is switching to SHA-256 in the near future.^{5</a>} Note that you don't need to specify the object's full name in these commands. Git can work with the prefix of a name, as long as there's no ambiguity: $ git show 1ed6 Some contents </code></pre> The index</code> file stores that state of the staging area. It tracks the state of files in your working tree, including the associated staged blob for each file, and some additional information like Unix file permissions. Git documentation often refers to the index and "staging area" interchangeably. You can inspect the staging area with the plumbing command $ git ls-files</code>: $ git ls-files --stage 100644 1ed6543483aafc93c5323daea1860bd7a29857d4 0 file1.txt </code></pre> Let's finalise our commit, attaching a description for this snapshot with the --message</code> option: $ git commit --message "Add file1.txt" [master (root-commit) 173bb18] Add file1.txt 1 file changed, 1 insertion(+) create mode 100644 file1.txt $ git status On branch master nothing to commit, working tree clean </code></pre> Let's run $ git show</code> to see the last commit that we just created: $ git show commit 173bb18cd1059b1efb048dc32442eb34b36c78a4 (HEAD -> master) Author: Avery <avery@example.com> Date: Fri Jan 1 00:00:00 2021 +0000 Add file1.txt diff --git a/file1.txt b/file1.txt new file mode 100644 index 0000000..1ed6543 --- /dev/null +++ b/file1.txt @@ -0,0 +1 @@ +Some contents </code></pre> Great! Git has created a commit object: named d8bc...</code></li> created by myself</li> at a certain date and time</li> containing a file called file1.txt</code> with the contents Some contents</code>.</li> </ul> Let's peek into the repository again to see what's changed: $ tree .git .git ├── branches ├── COMMIT_EDITMSG ├── config ├── HEAD ├── index ├── info │ └── exclude ├── logs │ ├── HEAD │ └── refs │ └── heads │ └── master ├── objects │ ├── 1e │ │ └── d6543483aafc93c5323daea1860bd7a29857d4 │ ├── 58 │ │ └── fa13f5e9dff3635df5401f0aa8ef5868f18e29 │ ├── fe │ │ └── 89dfe71214c0d45973c551c45a449b3b2f49f7 │ ├── info │ └── pack └── refs ├── heads │ └── master └── tags </code></pre> There's a lot more data in the repository now, so let's go through it. COMMIT_EDITMSG</code> is the file you edit to enter the commit message. It will be cleared for the next commit, but for now the message for the last commit remains. We can inspect the objects that were created with another Git plumbing command, $ git cat-file</code>: # Get the type (-t) of the object $ git cat-file -t fe89 tree # Print (-p) its contents $ git cat-file -p fe89 100644 blob 1ed6543483aafc93c5323daea1860bd7a29857d4 file1.txt </code></pre> Here we see Git has created a new type of object, a tree. Trees refer to a collection of blobs and other trees. For each blob entry, the tree stores: the name of the blob</li> the name of the file the blob stores the contents of</li> the associated Unix file permissions</li> </ul> For each subtree entry, the tree stores: the name of the tree</li> the name of the directory the tree stores the contents of</li> </ul> Thus trees represent a single directory, including its files and subdirectories. This tree in particular represents the root of our repository, which only contains a single file: file1.txt</code>. And now for the final object: $ git cat-file -t 58fa commit $ git cat-file -p 58fa tree fe89dfe71214c0d45973c551c45a449b3b2f49f7 author Avery <avery@example.com> 1615411472 +0000 committer Avery <avery@example.com> 1615411472 +0000 Add file1.txt </code></pre> As you might have noticed, this is the object referenced by the earlier $ git show</code>; it represents the new commit. Commit objects primarily contain: a reference to the tree for the root directory of this snapshot (i.e. the same as the root as the working tree)</li> the author and committer along with the date and time that each occurred</li> the commit message</li> a reference to the parent commit(s). This commit has no parents since it is the first commit in the repository</li> </ul> The author and committer are separate entries to support situations where one person creates a snapshot and shares it with someone else, who then creates a commit from it in their repository. Storing the author's identity keeps their name attributed to the snapshot. Git generated the commit's tree from the index. In this case, it's created a single tree for the root directory, which contains only file1.txt</code>. When there are staged files in subdirectories, index entries are collected together to form a tree for each directory. Each tree is linked to the trees that represent their subdirectories, ultimately forming the root tree which represents the full snapshot of the directory at the time of the commit. Other information in the commit can either come from you (i.e. the commit message), or be inferred, such as the author/committer's name and email, and the date and time of the commit. With commits, we have a way of tracking snapshots of the repository and a way of saving some contextual information about said snapshots, which serves the fundamental purpose of a version control system. The rest of Git is primarily for manipulating objects so that they can represent more than a simple linear history, and to support collaboration between multiple users. Normally you'll only be concerned with commits. You can run $ git log</code> to list the "current" commit followed by its ancestors. Right now it'll show the first commit, 173bb18</code>: $ git log commit 173bb18cd1059b1efb048dc32442eb34b36c78a4 (HEAD -> master) Author: Avery <avery@example.com> Date: Fri Jan 1 00:00:00 2021 +0000 Add file1.txt </code></pre> This commit is also our current HEAD, which means it will be a parent of the next commit that is created. Showing your changes</h3> $ git status</code> lists the files that have been changed between the previous commit and both the working tree and the current staging area. You can use $ git diff</code> to see a detailed list of the lines that have been changed in each file. Remember that, at this point, the staging area matches the previous commit: # Edit file1.txt $ echo "A new line" >> file1.txt # Create file2.txt $ echo "Some contents" > file2.txt $ git status On branch master Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: file1.txt Untracked files: (use "git add <file>..." to include in what will be committed) file2.txt no changes added to commit (use "git add" and/or "git commit -a") $ git diff diff --git a/file1.txt b/file1.txt index 1ed6543..ff709a8 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1,2 @@ Some contents +A new line </code></pre> The diff lists lines that have changed between two versions of the same file. New lines are prefixed with +</code>, and removed lines with -</code>. If multiple parts of a file have been changed, the diff will group them into distinct hunks, which encapsulate changes in nearby lines, with some unchanged lines for additional context. You can see that the diff references the staged blob for file1.txt</code>, 1ed6543</code>, as well as a new blob name ff709a8</code>. The new blob name is calculated from the file in the working tree, but is not actually saved as an object in the repository yet - a $ git add file1.txt</code> would result a blob with the name ff709a8...</code> being created. By default the diff shows changes between the staging area and the working tree, and excludes untracked files like file2.txt</code>. If we stage all changes, the staging area will match the working tree, and so a normal $ git diff</code> won't display anything: $ git add . $ git status On branch master Changes to be committed: (use "git restore --staged <file>..." to unstage) modified: file1.txt new file: file2.txt $ git diff # No difference between staging area and working tree, so no diff is shown </code></pre> We can instead use $ git diff --staged</code> to specify that we want to see the changes that we have staged. That means the differences between the last commit's tree and the staging area: $ git diff --staged diff --git a/file1.txt b/file1.txt index 1ed6543..ff709a8 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1,2 @@ Some contents +A new line diff --git a/file2.txt b/file2.txt new file mode 100644 index 0000000..1ed6543 --- /dev/null +++ b/file2.txt @@ -0,0 +1 @@ +Some contents </code></pre> If you were to create a commit right now and then run $ git show</code>, this same diff would be shown under the commit. Here's a graph to visualise the commands and the current states of the working tree, index, and the tree of the current HEAD. The red lines represent what the commands are comparing: Skipping the staging area</h3> If the staging area is inconvenient or unnecessary, you can specify the files you want to commit after $ git commit</code>, e.g. $ git commit file1.txt file2.txt</code>. This will immediately commit the contents of these files as they are in the working tree. Alternatively, run $ git commit --all</code>. This will immediately stage all tracked files as they currently are in the working tree, and commit them. Note that you still need to use $ git add</code> to track new files that you're committing for the first time, or list new files you want to track after --all</code>. Staging hunks within a file</h3> A common situation is having made several changes in your working tree and later realising you could split them up into several commits. E.g. you've fixed multiple bugs within a single source file. The staging area gives you the power to precisely control what goes into a snapshot, and this power makes it easier to keep commits self-contained. You can make sure each commit includes changes that serve a single purpose, that the project still builds correctly, and that all tests still succeed. If you have multiple changes in a single file and want to split them into different commits, you can stage only a subset of your changes with $ git add --patch</code>: $ echo -e "First line\nSome contents\nLast line" > file1.txt $ cat file1.txt First line Some contents Last line $ git add --patch file1.txt diff --git a/file1.txt b/file1.txt index 1ed6543..4a9a39f 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1,3 @@ +First line Some contents +Last line (1/1) Stage this hunk [y,n,q,a,d,s,e,?]? </code></pre> This interactive form goes through each hunk in your diff in the file(s) we want to stage, letting us choose what we want to do with it. If you ask for help with ?</code>, here are your options: y - stage this hunk n - do not stage this hunk q - quit; do not stage this hunk or any of the remaining ones a - stage this hunk and all later hunks in the file d - do not stage this hunk or any of the later hunks in the file s - split the current hunk into smaller hunks e - manually edit the current hunk ? - print help </code></pre> The most important ones are, unsurprisingly y</code> and n</code>, for "yes" and "no". s</code> will let you split a hunk up if it has unchanged lines between changes, which can be handy if Git hasn't managed to split the diff up the way you want it: (1/1) Stage this hunk [y,n,q,a,d,s,e,?]? s Split into 2 hunks. @@ -1 +1,2 @@ +First line Some contents (1/2) Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]? y @@ -1 +2,2 @@ Some contents +Last line (2/2) Stage this hunk [y,n,q,a,d,K,g,/,e,?]? y </code></pre> If you want to change the contents of the hunk, you can use e</code> to edit it manually: diff --git a/file1.txt b/file1.txt index 1ed6543..4a9a39f 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1,3 @@ +First line Some contents +Last line (1/1) Stage this hunk [y,n,q,a,d,s,e,?]? e </code></pre> The following will open in your editor. I've manually added the line containing +Another line</code>: # Manual hunk edit mode -- see bottom for a quick guide. @@ -1 +1,3 @@ +First line Some contents +Another line +Last line # --- # To remove '-' lines, make them ' ' lines (context). # To remove '+' lines, delete them. # Lines starting with # will be removed. # # If the patch applies cleanly, the edited hunk will immediately be # marked for staging. # If it does not apply cleanly, you will be given an opportunity to # edit again. If all lines of the hunk are removed, then the edit is # aborted and the hunk is left unchanged. </code></pre> After saving and quitting, you can see that your hunk has been staged without changing the file in your working tree: # The staged diff shows "Another line" that was added by editing the hunk $ git diff --staged diff --git a/file1.txt b/file1.txt index 1ed6543..77ef57a 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1,4 @@ +First line Some contents +Another line +Last line # Comparing the working tree to the staging area, "Another line" is shown as # removed, as editing the hunk didn't change the file in the working tree $ cat file1.txt First line Some contents Last line $ git diff diff --git a/file1.txt b/file1.txt index 1ed6543..77ef57a 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1,4 @@ First line Some contents -Another line Last line </code></pre> Restoring after changes</h3> If you've staged something and later decided you don't want to include it in the next commit, you can use $ git restore --staged</code> to reset the staging area to match HEAD: $ git status On branch master Changes to be committed: (use "git restore --staged <file>..." to unstage) modified: file1.txt Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: file1.txt $ git restore --staged . On branch master Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: file1.txt no changes added to commit (use "git add" and/or "git commit -a") </code></pre> Use the --patch</code> option to interactively pick which hunks to unstage, similar to $ git add</code>. You can use $ git restore</code> to restore the contents of the files in your working tree to match the staging area. Note that this will delete your changes from the working tree, which may not have been saved in the repository if you haven't staged or committed it. Finally, if you want to remove a file in a commit, you can remove the file from your working tree and subsequently $ git add</code> to stage the removal, or simply use $ git rm</code> to remove and stage in one command. refs: heads, tags and HEAD</h2> A ref is an alias for a specific commit or another ref, which are more user-friendly to use than full commit names. These are stored in files under .git/refs</code>, each containing either a full commit name or the name of another ref. They can be used in Git commands instead of commit names. Tags are user-specified refs, contained in .git/refs/tags</code>. They're typically used to mark significant commits with a user-friendly name, e.g. labelling the commit used for a software release with "v1.0". To create a new tag, use $ git tag</code>: $ git tag first $ cat .git/refs/tags/first 173bb18cd1059b1efb048dc32442eb34b36c78a4 $ git show first commit 173bb18cd1059b1efb048dc32442eb34b36c78a4 (HEAD -> master, tag: first) Author: Avery <avery@example.com> Date: Fri Jan 1 00:00:00 2021 +0000 Add file1.txt diff --git a/file1.txt b/file1.txt new file mode 100644 index 0000000..1ed6543 --- /dev/null +++ b/file1.txt @@ -0,0 +1 @@ +Some contents </code></pre> $ git show</code> lists the refs associated with a commit next to its name. In this case it's showing that refs HEAD</code>, master</code>, and the tag first</code> point to the first commit in the repository. A head (lowercase) points to a commit that is the "tip" of a branch. Branches represent a "line of development". When you add a commit while on a branch, Git automatically updates the head ref of a branch to point to the new commit, so the tip is maintained. "Branch" and "head" are sometimes used interchangeably in Git documentation. I'll expand on branches in the next section. Heads are stored in .git/refs/heads</code>. For example, heads/master</code> tracks Git's default branch, master</code>. Right now, it contains the name of the first commit object that was created. HEAD (uppercase) is a special ref that tells Git which commit will be the parent of the next commit. It will normally point to a head, e.g. master</code>, which is what the HEAD -> master</code> represents in the previous $ git show</code>: $ cat .git/HEAD ref: refs/heads/master </code></pre> So when HEAD is pointing to heads/master</code>, that means you are on branch master</code>, and creating a new commit will move heads/master</code> to point to it. When HEAD is pointing to a specific commit instead of a branch, this is known as being in a "detached HEAD" state. You can resolve this situation by manually manipulating your HEAD, which will be covered later. Most Git commands will default to using HEAD as their argument, including $ git tag</code>. You could specify any commit by name or by ref if you wanted, e.g. $ git tag d9bcd</code> or $ git tag HEAD</code>. Objects and refs are the two foundational components of Git: all operations involve manipulating some combination of the two. Branches</h2> The link between commits is that a commit refers to its parent(s) - this means a commit can have any number of "children". Through this, your commit history can diverge by having a commit with several children, and be rejoined later by a commit with several parents. These separate chains of commits are called branches, and they allow you to to create multiple chains of commits that exist in parallel. Git's default branch is called master</code>, but its name is configurable and so may vary between platforms and teams. Branching conventions also vary, so some repositories might use their default branch for active development, while others may only update it with each release to end-users. See extra resources</a> for some examples. You can add and list branches with $ git branch</code>, and switch which one you're with $ git switch</code>. Switching a branch will also update the index to match the target: # Create a new branch from the current HEAD, call it "my-branch" $ git branch my-branch $ git branch master my-branch # Branching has created a new head ref called my-branch $ tree .git/refs/heads .git/refs/heads/ ├── master └── my-branch # Both heads point to the first commit, "Add file1.txt" $ cat .git/refs/heads/ 173bb18cd1059b1efb048dc32442eb34b36c78a4 173bb18cd1059b1efb048dc32442eb34b36c78a4 # HEAD still points to master $ cat .git/HEAD ref: refs/heads/master # Switching updates HEAD to point to my-branch $ git switch my-branch Switched to branch 'my-branch' $ cat .git/HEAD ref: refs/heads/my-branch </code></pre> At this point we have a single commit, which is pointed to by the heads of both master</code> and my-branch</code> and the tag first</code>. It is also currently our HEAD. The above diagram represents the current commit history of the repository - in this case, just a single box for the first commit. The labels without boxes represent the current refs: the heads for master</code> and the new branch my-branch</code>, the tag first</code>, and finally HEAD</code>. These point to the contents of the ref, i.e. another ref or a commit. Let's add a new commit while we're on my-branch</code>: # Use the --allow-empty option so we don't need to commit any changes $ git commit --allow-empty --message "Commit on my-branch" [my-branch 630d4e3] Commit on my-branch # HEAD still points to my-branch $ cat .git/HEAD ref: refs/heads/my-branch # master still points to the first commit, "Add file1.txt" $ cat .git/refs/heads/master 173bb18cd1059b1efb048dc32442eb34b36c78a4 # my-branch now points to the new commit, "Commit on my-branch" $ cat .git/refs/heads/my-branch 630d4e31ff361999100a45bd38005bc01ac2f935 # List HEAD followed by its chain of ancestors # --oneline gives us a single-line summary $ git log --oneline --all 630d4e3 (HEAD -> my-branch) Commit on my-branch 173bb18 (tag: first, master) Add file1.txt </code></pre> Now we've got a new commit, 630d4e3</code>, whose parent is the first commit, 173bb18</code>. Since HEAD was pointing to heads/my-branch</code>, Git updated this head to point to the new commit, so it continues to track the tip of the branch. Note that heads/master</code> and tags/first</code> both continue to point to the original commit 173bb18</code>, and HEAD</code> still points to heads/my-branch</code>: Note that in the diagram, 630d4e3</code> points to its parent - it is the child commit which refers to its parent(s), not the other way around. Let's switch back to master</code> and create another commit: $ git switch master Switched to branch 'master' $ git commit --allow-empty --message "Commit on master" [master 0538d0d] Commit on master # --all includes the tips of all branches # --graph visualises ancestry: commits are asterisks, lines show parents $ git log --oneline --all --graph 0538d0d (HEAD -> master) Commit on master | 630d4e3 (my-branch) Commit on my-branch |/ 173bb18 (tag: first) Add file1.txt </code></pre> At this point, master</code> and my-branch</code> have diverged: they both contain commits that the other does not. Branches are useful for separating changes that are a work-in-progress, like implementing a new feature to your application. Working on a separate branch avoids disrupting others with your potentially broken changes, and also avoids collisions between branches until you're ready to merge your changes. Merge commits</h3> Once you're happy with the state of your branch and want to include your changes in another branch, you can rejoin your branch to master</code> (or any other). This is done with $ git merge</code>, which will create a new commit with multiple parents, combining the changes in all of them: $ git log --oneline --graph 0538d0d (HEAD -> master) Commit on master 173bb18 (tag: first) Add file1.txt # The merge command shares several options with commit, including --message $ git merge my-branch --message "Merge my-branch into master" Already up to date. Merge made by the 'recursive' strategy. # The new commit has two parents: $ git cat-file -p HEAD tree fe89dfe71214c0d45973c551c45a449b3b2f49f7 parent 0538d0d4729a768be286d2b6b6a00fe7c7211d94 parent 630d4e31ff361999100a45bd38005bc01ac2f935 author Avery <avery@example.com> 1609459200 +0000 committer Avery <avery@example.com> 1609459200 +0000 Merge my-branch into master # Now "Commit on my-branch" also appears in the history of master $ git log --oneline --graph 031b880 (HEAD -> master) Merge my-branch into master |\ | 630d4e3 (my-branch) Commit on my-branch | 0538d0d Commit on master |/ 173bb18 Add file1.txt </code></pre> After merging, master</code> has another new commit f0d4</code>, which has two parents from different branches: 6679</code> from my-branch</code> and d86e</code> from master</code>. This single merges the commits from my-branch</code> into master</code> - such commits are called merge commits. The merge commit message comes from the message specified in the $ git merge</code> command. The head of my-branch</code> remains pointed to 6679</code>, as the merge commit was made on the master</code> branch, so we say that my-branch</code> has been merged into master</code>. At this point you could continue to add new commits on my-branch</code> and master</code>, and they could be merged together again later. At this point you can delete your branch: $ git branch --delete my-branch Deleted branch my-branch (was 630d4e3). $ git log --oneline --graph 031b880 (HEAD -> master) Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 Add file1.txt </code></pre> Notice that the log still displays the separate branches in history - this is thanks to the merge commit. Git traverses both parents of the merge to show the point at which master</code> and the other branch diverged, and shows that they were merged again. However, the head for my-branch</code> has been removed. Fast-forward merging</h3> Sometimes a merge commit isn't necessary - for example, when merging branches that haven't diverged. In such cases, Git can simply adjust the head of the destination branch to match the branch being merged. These merges are known as fast-forwards, and Git can automatically detect when it's possible and apply this when you run $ git merge</code>. For an example, let's create a new commit on another branch: # Create my-branch again and switch to it $ git switch --create my-branch Switched to a new branch 'my-branch' $ git commit --allow-empty --message "Fast-forward commit" [my-branch 10498bd] Fast-forward commit # Switch back to master $ git switch master Switched to branch 'master' </code></pre> Now we have a single new commit on my-branch</code>, pointing to the current head of master</code>, meaning these two branches have not diverged. Let's merge my-branch</code> into master</code> once more: $ git merge my-branch Updating 031b880..10498bd Fast-forward $ git log --oneline --graph 10498bd (HEAD -> master, my-branch) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Git has automatically determined that the branches haven't diverged, and so a fast-forward was performed instead of creating a merge commit. $ git merge</code> lets you control its strategy with a few options: --ff</code>: Create a merge commit if the branches have diverged, otherwise fast-forward. This is the default behaviour.</li> --no-ff</code>: Always create a merge commit.</li> --ff-only</code>: Only fast-forward. If the branches have diverged, the merge fails.</li> </ul> Finally, let's delete my-branch</code> again, since we're finished with it: # Delete my-branch again $ git branch --delete my-branch Deleted branch my-branch (was 10498bd). </code></pre> Fast-forwarding saves us from creating merge commits when they're unnecessary, which can help keep the commit history tidy. However, without a merge commit to indicate that a merge has taken place, the fact the merged branch existed is hidden. This may or may not be desirable, so it's worth thinking about when deciding between a normal merge and a fast-forward: Cherry-picking commits</h3> If there's a single commit in another branch that you want to incorporate into your branch, you can copy it using $ git cherry-pick</code>. This will take the diff of the target commit (i.e. only the changes introduced in that commit) and apply it to your current working tree, then create a new commit copying the target's message and author. Take a situation where you have a number of commits in another branch: # Create a new branch $ git switch --create cherrypick-from Switched to a new branch 'cherrypick-from' # Create two commits, the first one is empty $ git commit --allow-empty --message "First cherry-pick commit" [cherrypick-from 9147754] First cherry-pick commit # Whilst the second one adds a new file called file2.txt $ echo "Some contents" > file2.txt $ git add file2.txt $ git commit --message "Second cherry-pick commit" [cherrypick-from cb3c9c2] Second cherry-pick commit 1 file changed, 1 insertion(+) create mode 100644 file2.txt # Resulting in the following history: $ git log --oneline --graph cb3c9c2 (HEAD -> cherrypick-from) Second cherry-pick commit 9147754 First cherry-pick commit 10498bd (master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Now we have a couple of commits that are only on branch cherrypick-from</code>. If we want to include only one of these commits in another branch, we can cherry-pick to copy it. For example, let's create a new branch based on master</code> and copy the "Second cherry-pick commit" commit which creates file2.txt</code>: # Create and switch another branch based off master, called cherrypick-to $ git switch --create cherrypick-to master Switched to branch 'cherrypick-to' # Use cherry-pick on the head of cherrypick-from # i.e. cb3c9c2: "Second cherry-pick commit" $ git cherry-pick cherrypick-from [cherrypick-to a9923b0] Second cherry-pick commit Date: Fri Jan 1 00:00:00 2021 +0000 1 file changed, 1 insertion(+) create mode 100644 file2.txt $ cat file2.txt Some contents $ git log --oneline --graph --all cb3c9c2 (cherrypick-from) Second cherry-pick commit 9147754 First cherry-pick commit | a9923b0 (HEAD -> cherrypick-to) Second cherry-pick commit |/ 10498bd (master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Now a "Second cherry-pick commit" commit also exists cherrypick-to</code>, which has added the new file2.txt</code>. Note that the cherry-picked commit has a different hash from the original, indicating that they are distinct from one another. The parent of the new commit is different - even if everything else in the commit was copied exactly, the object's hash would differ and thus the commits would be different objects. # Clean up: delete the cherry-pick branches $ git switch master # Use the --force option to delete unmerged branches, which will cause the new # "Cherry-pick" commits to be lost $ git branch --delete --force cherrypick-from cherrypick-to Deleted branch cherrypick-from (was 9147754). Deleted branch cherrypick-to (was a9923b0). $ git log --oneline --graph --all 10498bd (HEAD -> master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Solving conflicts</h3> A conflict is an error that can occur when Git attempts to merge changes from commits that have different ancestries. Specifically, they occur when two diverged branches have applied different changes to the same part of the same file: # Create a new branch and create a commit that file1.txt $ git switch --create conflict Switched to a new branch 'conflict' $ echo "Changes from the conflict branch" > file1.txt $ git commit --all --message "Commit on conflict branch" [conflict 9cbabbf] Commit on conflict branch 1 file changed, 1 insertion(+), 1 deletion(-) # Switch back to master and add a commit that modifies file1.txt $ git switch master Switched to branch 'master' $ echo "Changes from master" > file1.txt $ git commit --all --message "Conflict on master" [master 3b9e2b8] Conflict on master 1 file changed, 1 insertion(+), 1 deletion(-) # Attempt to merge in the conflict branch $ git merge conflict Auto-merging file1.txt CONFLICT (content): Merge conflict in file1.txt Automatic merge failed; fix conflict and then commit the result. </code></pre> Git pauses during the merge to alert us - it has tried to automatically merge the changes that branch conflict</code> has made to file1.txt</code>, but failed. This is what we call a conflict. If we check $ git status</code>, we can see that we are in a new state with "unmerged" files: $ git status On branch master You have unmerged paths. (fix conflict and run "git commit") (use "git merge --abort" to abort the merge) Unmerged paths: (use "git add <file>..." to mark resolution) both modified: file1.txt no changes added to commit (use "git add" and/or "git commit -a") </code></pre> As the status suggests, if you don't want to go through with the merge at this point, you can run $ git merge --abort</code> to cancel it. Unmerged files are tracked within the index, which now associates three different blobs for file1.txt</code>: $ git ls-files --stage 100644 1ed6543483aafc93c5323daea1860bd7a29857d4 1 file1.txt 100644 4c573990878e6ae6f463ce0ab81b6eac635af26b 2 file1.txt 100644 a4e9201a54a9db3b35b39a09ded144328e5c837a 3 file1.txt $ git show 1ed6543483aafc93c5323daea1860bd7a29857d4 Some contents $ git show 4c573990878e6ae6f463ce0ab81b6eac635af26b Changes from master $ git show a4e9201a54a9db3b35b39a09ded144328e5c837a Changes from the conflict branch </code></pre> The first blob is the original contents of file1.txt</code>, before either branch applied the changes that are causing the conflict. The second is the contents of the file in the target branch, master</code>, and the third from branch conflict</code>. Inside the repository, there are also a number of files that Git uses to track information about the current merge: # MERGE_HEAD, MERGE_MODE, MERGE_MSG, and ORIG_HEAD have now appeared: $ tree -L 1 .git .git ├── branches ├── COMMIT_EDITMSG ├── config ├── HEAD ├── index ├── info ├── logs ├── MERGE_HEAD ├── MERGE_MODE ├── MERGE_MSG ├── objects ├── ORIG_HEAD └── refs </code></pre> MERGE_HEAD</code> contains the name of the commit currently being merged into the target branch. In this case, it contains the name of the tip of conflict</code>. Like HEAD</code>, despite not being stored in .git/refs</code>, MERGE_HEAD</code> is a valid ref: $ git show MERGE_HEAD commit 9cbabbfdeefd8189cf63e01b40697a9063de519e (conflict) Author: Avery <avery@example.com> Date: Sat Apr 17 12:13:11 2021 +0100 Commit on conflict branch diff --git a/file1.txt b/file1.txt index 1ed6543..a4e9201 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1 @@ -Some contents +Changes from the conflict branch </code></pre> ORIG_HEAD</code> contains the target commit which MERGE_HEAD</code> is being merged into. In this case, the it contains the name of the tip of master</code> at the time the merge started. This is the commit that you will be returned to if you cancel the merge: $ git show ORIG_HEAD commit 95f682d4a83d7fd55f35dc09e9db6e403639b81c (HEAD -> master) Author: Avery <avery@example.com> Date: Sat Apr 17 12:14:18 2021 +0100 Conflict on master diff --git a/file1.txt b/file1.txt index 1ed6543..919d56c 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1 @@ -Some contents +Changes from master </code></pre> Both of these commits will be used as parents of the merge commit once all conflicts have been resolved. MERGE_MSG</code> contains the message that will be used for the merge commit, and MERGE_MODE</code> contains the merge strategy being used (e.g. no-ff</code>). In this unmerged state, file1.txt</code> contains a combination of the conflicting changes: <<<<<<< HEAD Changes from master ======= Changes from the conflict branch >>>>>>> conflict </code></pre> The lines beginning with <</code>, =</code>, and ></code> are called conflict markers, and indicate the areas of the file in which conflicts have occurred. The first section of a conflict area is between the begin marker line <<<</code> and ===</code>, and is labelled with the ref or commit (in this case, HEAD</code>). This label indicates where those changes have come from. Similarly, the lines between ===</code> and the ending marker >>></code> are the contents of the other ref or commit in this conflict, in this case the tip of conflict</code>. Generally, "our" changes are in the first section, and "their" changes are in the second section. In a merge, "our" changes are from the current branch which has changes being merged into, and "their" changes are from the branch that is being merged from. To resolve the conflict, replace the conflict markers with the contents that should be there after the merge process. This can be the contents from either marker region, or some combination of the two. For this example, I'm going to change the line entirely: $ echo "Changes after merging" > file1.txt # Stage file1.txt to mark it as merged $ git add file1.txt $ git status On branch master All conflicts fixed but you are still merging. (use "git commit" to conclude merge) $ git commit </code></pre> You'll be prompted for a commit message as usual: Merge branch 'conflict' # Conflicts: # file1.txt # # It looks like you may be committing a merge. # If this is not correct, please run # git update-ref -d MERGE_HEAD # and try again. # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # # On branch master # All conflicts fixed but you are still merging. # </code></pre> After you save and quit your editor, the merge commit will be created: [master 7c5b805] Merge branch 'conflict' $ git show commit 7c5b8052af615dfc4a672ca4fe937f29db8c1348 (HEAD -> master) Merge: 3b9e2b8 9cbabbf Author: Avery <avery@example.com> Date: Fri Jan 1 00:00:00 2021 +0000 Merge branch 'conflict' diff --cc file1.txt index 4c57399,a4e9201..7c35f44 --- a/file1.txt +++ b/file1.txt @@@ -1,1 -1,1 +1,1 @@@ - Changes from master -Changes from the conflict branch ++Changes after merging </code></pre> Instead of showing just a simple before-and-after, the diff shows the changes applied to both the contents from master</code> and conflict</code>, followed by the new change which replaced both. This is a combined diff, showing the change that the new commit introduces in comparison to all of its parents.^{6</a> If we kept the contents from either master</code> or conflict</code> the diff would be empty; technically the merge would have introduced no new changes. In that case, you could instead use $ git diff <merge commit>..<merge-commit>^</code> to show changes that the merge introduced into the target branch. I'll explain this syntax in the next section.} # Delete the new branch to clean up $ git branch --delete conflict Deleted branch conflict (was 9cbabbf). </code></pre> Exploring history and branches</h2> Resetting HEAD</h3> $ git reset</code> changes the current HEAD (or the current branch's head) to the specified target, and optionally affects the staging area and working tree. Manually changing refs is handy in situations where you want to manipulate history to, for example, undo a commit, or to update a branch to match another branch. To undo the commits made during the conflicts example, we can reset back to 10498bd</code>, "Fast-forward commit": $ git log --oneline --graph 7c5b805 (HEAD -> master) Merge branch 'conflict' |\ | 9cbabbf Commit on conflict branch | 3b9e2b8 Conflict on master |/ 10498bd Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt $ git reset 10498bd # "Fast-forward commit" Unstaged changes after reset: M file1.txt $ git log --oneline --graph 10498bd (HEAD -> master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> By default, $ git reset</code> won't affect the working tree, so file1.txt</code> keeps the contents it had in 7c5b805</code>. This is called a "mixed" reset. Git lists file1.txt</code> under "Unstaged changes after reset" to indicate this file doesn't match the staging area after the reset. If want to reset HEAD but also leave the staging area as it is, you can use the --soft</code> option: $ cat file1.txt Changes from the conflict branch # A soft reset won't affect the staging area either $ git add file1.txt $ git reset --soft 95f682d # "Commit on master branch" # The staging area keeps the changes that were staged in file1.txt $ git diff --staged git diff --staged diff --git a/file1.txt b/file1.txt index 919d56c..a4e9201 100644 --- a/file1.txt +++ b/file1.txt @@ -1 +1 @@ -Changes from the master branch +Changes after merging </code></pre> Finally, a hard reset using --hard</code> will set everything to match the target: HEAD, the staging area, and the working tree: $ git reset --hard 10498bd # "Fast-forward commit" $ cat file1.txt Some contents $ git status On branch master nothing to commit, working tree clean $ git log --oneline --graph 10498bd (HEAD -> master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> And now we're back to "Fast-forward commit", as if the conflict merging had never happened! Referring to commits by ancestry</h3> You can also refer to commits from their descendants in a relative manner, which saves you the hassle of searching for names when dealing with the recent ancestors of a ref: # "The parent of HEAD" $ git show --oneline HEAD^ 031b880 Merge my-branch into master # "The grandparent of HEAD" $ git show --oneline HEAD^^ 0538d0d Commit on master </code></pre> The ^</code> (caret) character means "parent of", so repeating the character will traverse as many commits as you want. Note how commits with multiple parents are handled: HEAD^^</code> shows "Commit on master", despite the merge commit also having "Commit on my-branch" as a parent. This is because the first parent is implicitly chosen - for merge commits this is a commit on the branch that was merged into. So ^</code> really means first parent. To manually choose which parent you want, use a number following the ^</code> (e.g. ^2</code> means "second parent"): # "Second parent of the parent of HEAD" $ git show --oneline HEAD^^2 630d4e3 Commit on my-branch </code></pre> You can keep adding ^</code> characters afterwards to traverse more parent commits. Instead of repeating ^</code> for each parent, you can use ~</code> (tilde), which always means "first parent". Appending a number instead specifies how many parents you want to traverse, and so is equivalent to using ^</code> sequentially that number of times: # "The first grandparent of HEAD" $ git show --oneline HEAD^^ 0538d0d Commit on master # "The grandparent of HEAD" again $ git show --oneline HEAD~2 0538d0d Commit on master </code></pre> You can combine both systems to get to any commit you want: # Traverse the second parent of the merge commit to get the first commit # First parent -> second parent -> first parent $ git show --oneline --no-patch HEAD~^2^ 173bb18 (tag: first) Add file1.txt # The same path: ^ and ~ can be substituted if no numbers are appended $ git show --oneline --no-patch HEAD^^2~ 173bb18 (tag: first) Add file1.txt </code></pre> Garbage collection and the reflog</h3> When using $ git reset</code> and other commands that affect refs, you can enter situations where commits are no longer accessible from any ref. But, as demonstrated earlier, you can still access these unreachable commits and reset to them. They aren't immediately deleted, so there's no need to worry about immediately losing your work. If you've lost and forgotten the name of a commit, you'll be able to find it using the reflog ("ref-log"), which keeps a local history of changes to refs. These are stored in .git/logs</code>. $ git reflog 10498bd (HEAD -> master) HEAD@{0}: reset: moving to 10498bd 173bb18 (tag: first) HEAD@{1}: reset: moving to HEAD 173bb18 (tag: first) HEAD@{2}: reset: moving to HEAD 173bb18 (tag: first) HEAD@{3}: reset: moving to first 10498bd (HEAD -> master) HEAD@{4}: commit: Fast-forward commit ... </code></pre> In this case, the last few entires in the reflog are showing the most recent commits and the resets from the previous section. The reflog introduces another relative name syntax, specifically for the previous states of refs. This follows the format ref@{offset}</code>, with older entries using larger offsets: # HEAD@{0} is equivalent to HEAD $ git show --oneline HEAD@{0} 10498bd (HEAD -> master) Fast-forward commit $ git show --oneline HEAD@{1} 173bb18 (tag: first) Add file1.txt </code></pre> A reflog is kept for each head - so as long as an unreachable commit is accessible through commits in any reflog Git will keep them around. Reflogs are pruned over time when you run Git commands, by default keeping the last 90 days of history,^{7</a> which is plenty of time in most cases to recover useful work that has been accidentally made unreachable. Unreachable objects are created fairly regularly while using Git (e.g. when you stage a file multiple times before committing it) - garbage collection stops these objects from bloating your local repository.} Don't rely on the reflog to save things that should be kept safe in the long-term - use branches</a> or the stash</a> instead. Stashing changes</h3> A common situation is to have some additional changes after a commit, but these changes may affect test results, which makes it harder to check your commit stands on its own. Alternatively, you might want to quickly switch to working on something else, starting from a clean slate whilst saving your existing changes. Instead of creating a feature branch or temporary commit that you'll forget about, you can use the stash: $ echo "something" >> file1.txt $ git status On branch master Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: file1.txt $ git stash Saved working directory and index state WIP on master: 10498bd Fast-forward commit $ cat file1.txt Some contents $ git stash list stash@{0}: WIP on master: 10498bd Fast-forward commit </code></pre> Git has added a stash</code> ref that points to a commit: $ cat .git/refs/stash 77a35513527e68bdd2adef9a9278608cda19e6a5 $ git show stash commit 77a35513527e68bdd2adef9a9278608cda19e6a5 (refs/stash) Merge: 10498bd 943a226 Author: Avery <avery@example.com> Date: Sat Mar 27 21:40:01 2021 +0000 WIP on master: 10498bd Fast-forward commit diff --cc file1.txt index 1ed6543,1ed6543..29b2192 --- a/file1.txt +++ b/file1.txt @@@ -1,1 -1,1 +1,2 @@@ Some contents ++something # The stash commit is a merge commit, merging another commit into the current # head of master. Here's the commit it's merging: $ git show 943a226 commit 943a226ff623b6a077084ff09699234903d80a84 Author: Avery <avery@example.com> Date: Fri Jan 1 00:00:00 2021 +0000 index on master: 10498bd Fast-forward commit $ git log --oneline --graph stash 77a3551 (refs/stash) WIP on master: 10498bd Fast-forward commit |\ | 943a226 index on master: 10498bd Fast-forward commit |/ 10498bd (HEAD -> master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Stashing creates two commits: "index on...", which contains only changes that were staged, and "WIP on...", which contains the remaining changes you had in the working tree. The HEAD at the time of the stash is used a parent for both commits, with its name being appended to the message of the stash commits. The "WIP" commit is a merge commit, merging the "index" commit into the HEAD, so it includes both the staged and unstaged changes. Finally, refs/stash</code> points to the "WIP" commit. You can list your saved stashes with $ git stash list</code>: $ git stash --list stash@{0}: WIP on master: 10498bd Fast-forward commit </code></pre> The list uses the syntax for ref history, which hints that the stash system relies on the reflog to keep track of stash commits. Use $ git reflog</code> to inspect the reflog in .git/logs/refs/stash</code>: $ git reflog stash 77a3551 (refs/stash) stash@{0}: WIP on master: 10498bd Fast-forward commit </code></pre> Older stash commits are listed later in the reflog, and have a larger index. Unlike other reflogs, Git does not garbage collect old entries in the stash reflog (by default), so it's safe to save changes there in the long term. Use $ git stash apply</code> to apply (i.e. unstash) the "WIP" commit, placing its changes back into the working tree. In this example we only had unstaged changes, but stash apply</code> would also restore changes to the staging area if there were any: $ git stash apply On branch master Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: file1.txt no changes added to commit (use "git add" and/or "git commit -a") $ git stash list stash@{0}: WIP on master: 10498bd Fast-forward commit </code></pre> After applying it, the stash entry is kept in the log, in case you want to keep it around. You can use $ git stash drop</code> to remove the top stash commit from history, without applying it. Alternatively, you can drop a specific stash by name or index, or use $ git stash clear</code> to remove all of them: # Create a new stash, using the --message option to set the stash commit # message $ echo "something" >> file1.txt $ git stash --message "Stash 0" Saved working directory and index state On master: Stash 0 $ git stash list stash@{0}: On master: Stash 0 stash@{1}: WIP on master: 10498bd Fast-forward commit $ git stash drop 1 Dropped refs/stash@{1} (77a35513527e68bdd2adef9a9278608cda19e6a5) $ git stash list stash@{0}: On master: Stash 0 $ git stash clear $ git stash list # No stashes remain $ git reflog stash # The stash reflog has been cleared </code></pre> To apply and drop a stash in one command, use $ git stash pop</code>: # Create another stash # $ git stash push is the same as $ git stash $ echo "something" >> file1.txt $ git stash push Saved working directory and index state WIP on master: 10498bd Fast-forward commit $ git stash pop On branch master Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: file1.txt no changes added to commit (use "git add" and/or "git commit -a") Dropped refs/stash@{0} (77a35513527e68bdd2adef9a9278608cda19e6a5) $ git stash list # No stashes remain </code></pre> Use the --index</code> option with pop</code> or apply</code> to only apply the changes in the "index" commit, i.e. the changes that were staged before the stash. Instead of stashing all changes in your working tree, you can pick which ones you want by providing a list of files after a --</code> (double-dash) to separate them from the options: $ git stash push -- file1.txt file2.txt</code>. Use the --patch</code> option with $ git stash push</code> to interactively select which changes you want to stash, similarly to $ git add --patch</code>. By default, stashing won't include untracked files - if you want to include them, use the --include-untracked</code> option with $ git stash push</code>. Some Git commands like $ git merge</code> have an --autostash</code> option, which is handy if your working tree is dirty when you want to do a merge. This option stashes your changes before the merge, then applies the stash after it. Rewriting history</h2> Commit objects are immutable, meaning the name of a commit depends on the contents of all the files in the repository, the date and time of the commit, and several other factors. If we want to edit commits in the repository, we can't simply change the existing commits. We can however create new commits based on some existing ones and replace the existing commits, effectively rewriting history. This is handy for correcting mistakes and keeping commits self-contained. Since all commits are performed on your local repository rather than relying on an external server, you're free to rewrite history before you're happy to share it with others. You can also rewrite history after sharing your commits, but this is a more dangerous operation. Undoing commits</h3> If you simply want to get rid of the commit on the tip of the branch, you can use $ git reset</code> (as explained in the reflog</a> section). Amending commits</h3> The simplest scenario is wanting to edit the commit on the tip of the branch, (i.e. the most recent one in history) - $ git commit</code> has the option --amend</code> which you can use in this scenario. Instead of creating a new commit, amending a commit will remove the last commit, then use your currently staged contents to create a new commit object. You'll be presented with the previous commit's message to edit: $ rm file1.txt $ git commit --all --amend </code></pre> You'll then be presented with this in your editor: Fast-forward commit # Please enter the commit message for your changes. Lines starting # with '#' will be ignored, and an empty message aborts the commit. # # Date: Fri Jan 1 00:00:00 2021 +0000 # # On branch master # Changes to be committed: # deleted: file1.txt # </code></pre> After saving and quitting, your amend will be applied: [master 33cda34] Fast-forward commit Date: Fri Jan 1 00:00:00 2021 +0000 1 file changed, 1 deletion(-) delete mode 100644 file1.txt $ git log --oneline --graph 33cda34 (HEAD -> master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt # 33cda34 has replaced 10498bd $ git cat-file -p 33cda34 tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904 parent 031b880dda2a0afa441418fb494c598900092d22 author Avery <avery@example.com> 1609459200 +0000 committer Avery <avery@example.com> 1609459200 +0000 Fast-forward commit # Undo the amend by resetting back to 10498bd # You could also reset to HEAD@{1}, which will also point to 10498bd $ git reset --hard 10498bd HEAD is now at 10498bd Fast-forward commit </code></pre> The amended commit keeps the author from the original commit, but the committer will store new information, i.e. who amended the commit, and when. Rebasing</h3> If your situation involves something other than the commit at the tip of a branch, $ git rebase</code> provides much more flexibility (and complexity). Rebasing is the process of taking a contiguous set of commits and reapplying each to create a new set. It can reapply those commits on top of a different "tip", changing the commit that the set is attached to - think of this as "re-parenting". You can use $ git rebase</code> with another branch as the target (known as the new base), which will take the commits on the current branch that aren't in the target branch and reapply them to the current tip of the target. In this case, "reapplying" a commit effectively means cherry-picking</a> it. In the end, your branch will be reformed into one you can merge with a fast-forward</a>. To demonstrate: # Create a new branch from the first commit, using the tag 'first' $ git switch --create rebasing first Switched to a new branch 'rebasing' $ git commit --allow-empty --message "Rebase commit" [rebasing 3c97d1a] Rebase commit $ git log --oneline --graph 3c97d1a (HEAD -> rebasing) Rebase commit 173bb18 (tag: first) Add file1.txt $ git log --oneline --graph --all 33cda34 (master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ | 3c97d1a (HEAD -> rebasing) Rebase commit |/ 173bb18 (tag: first) Add file1.txt </code></pre> Here's a clearer visualisation of the commit graph: Now three commits are parented to the first commit, including the new commit we just made. Now let's try rebasing: $ git rebase master Successfully rebased and updated refs/heads/rebasing. $ git log --oneline --graph --all 3ce5c9d (HEAD -> rebasing) Rebase commit 10498bd (master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Now our new commit is parented to the tip of master</code> instead of the first commit: # Delete the branch to clean this example up # Use the --force option since there are unmerged commits # Commit 3ce5c9d will become unreachable $ git switch master $ git branch --delete --force rebasing Deleted branch rebasing (was 3ce5c9d). </code></pre> Note that conflicts</a> can occur while commits are being reapplied. In that case, Git will inform you and let you interactively resolve the conflict</a> similarly to a merge. Use: $ git rebase --abort</code> to cancel the rebase entirely, and restore the branch to its state before the rebase started</li> $ git rebase --skip</code> to skip the conflicting commit and resume the rebase</li> $ git rebase --continue</code> after the conflict has been resolved to amend the conflicting commit and resume the rebase</li> </ul> Git will fast-forward (i.e. skip over) commits that don't need to be rewritten, which saves you from accidentally rewriting commits that you shouldn't. You can use the --no-ff</code> or --force-rebase</code> options to explicitly disable this behaviour. Like $ git merge</code>, $ git rebase</code> has an --autostash</code> option to quickly stash away changes in your working tree before rebasing, and reapply them after the rebase is complete. Interactive rebasing</h3> You can manually influence the rebase process, which makes a rebasing much more powerful tool than simply copying a set of commits to a new tip. Rebase using the --interactive</code> option and Git will open your editor, presenting you with the list of commits that ill be affected. You can edit the list to specify what to do with each commit: $ git rebase --interactive first </code></pre> The following is opened in your editor: pick 0538d0d Commit on master # empty pick 630d4e3 Commit on my-branch # empty pick 10498bd Fast-forward commit # empty # Rebase 173bb18..33cda34 onto 173bb18 (3 commands) # # Commands: # p, pick <commit> = use commit # r, reword <commit> = use commit, but edit the commit message # e, edit <commit> = use commit, but stop for amending # s, squash <commit> = use commit, but meld into previous commit # f, fixup <commit> = like "squash", but discard this commit's log message # x, exec <command> = run command (the rest of the line) using shell # b, break = stop here (continue rebase later with 'git rebase --continue') # d, drop <commit> = remove commit # l, label <label> = label current HEAD with a name # t, reset <label> = reset HEAD to a label # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>] # . create a merge commit using the original merge commit's # . message (or the oneline, if no original merge commit was # . specified). Use -c <commit> to reword the commit message. # # These lines can be re-ordered; they are executed from top to bottom. # # If you remove a line here THAT COMMIT WILL BE LOST. # # However, if you remove everything, the rebase will be aborted. # </code></pre> Each line is a command, the first word being the command itself and rest are are arguments. If the argument is a single commit, its name can be followed by any text, which Git uses to show the first line of the commit message. Lines starting with a #</code> are comments which won't be interpreted as a command. The comments below the list give you a quick tutorial on the available commands. pick</code> is the most basic command, which means to cherry-pick the specified commit. Commands are executed from top to bottom, so reorder the pick</code> commands if you want to change the order of the new commits. Place a squash</code> command immediately after the commit that the argument should be combined into. Once you save and quit, Git will run through each command in sequence and you'll be left with a history rewritten exactly the way you want it. The rebase process may be interrupted if you use an interactive command like edit</code> or break</code>, or if there are any conflicts</a>. In this case you can use the --continue</code> option once you're done. Squash and fixup commits</h3> If you simply want to amend a commit that isn't at the tip of the branch, you can use $ git commit --squash <commit></code> and $ git commit --fixup <commit></code>, which both interact with the rebase process. The new commit's message will be prepended with "squash!" or "fixup!" and the first line of the target commit's message. If you then run $ git rebase --interactive --autosquash <commit></code> (where <commit></code> is the earliest commit you want to amend), the list of commands will appropriately reorder and replace the command for squash and fixup commits. This will save you the trouble of manually editing the command list in one go, instead setting up the commands as you commit. For example: $ git commit --allow-empty --fixup first [master 4e04c03] fixup! Add file1.txt # Use --root to include root (i.e. first) commit in the rebase $ git rebase --interactive --autosquash --root </code></pre> You'll be presented with the following command list: pick 173bb18 Add file1.txt fixup 4e04c03 fixup! Add file1.txt # empty pick 0538d0d Commit on master # empty pick 630d4e3 Commit on my-branch # empty pick 10498bd Fast-forward commit # empty </code></pre> In this case the first commit will be amended, and our commit that merged commits 2 and 3 will be skipped. You can run $ git config --global rebase.autoSquash true</code> to enable the --autosquash</code> option by default. Remotes</h2> Everything discussed so far are things you can do on your local repository - the final piece of the Git puzzle is how to collaborate with others by sharing your commits. Git can keep track of remote repositories, which are normal repositories that exist at some location outside your .git</code> repository. These can be elsewhere on your local system, or over the local network/internet using protocols like HTTP(S) or SSH. Creating a new remote</h3> To create a remote repository for our current repository, we can use $ git clone</code>. For example, if your working tree exists in the subdirectory repository</code>: $ ls repository $ git clone --bare repository remote Cloning into bare repository 'remote'... done. $ ls repository remote $ tree -L 1 remote remote/ ├── branches ├── config ├── description ├── HEAD ├── hooks ├── info ├── objects ├── packed-refs └── refs </code></pre> Cloning copies a repository into another location, in this case repository/.git</code> into the directory remote</code>. With the --bare</code> option, Git will make the target directory a plain repository without a working tree, so now remote</code> mirrors repository/.git</code>, including refs and reachable objects only. To instead set up an empty repository for a remote, you can run $ git init --bare</code>. Similarly to $ git clone --bare</code>, this repository won't have a working tree associated with it. You can still use a few normal Git commands on a bare repository: $ cd remote $ git log --oneline --graph --all 10498bd (HEAD -> master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt # Without a working tree most operations will fail $ git switch --create test fatal: this operation must be run in a work tree </code></pre> Back in our normal repository, we can register the new repository as our primary remote, called origin</code>: $ cd ../repository $ git remote add origin ../remote # A new section has been added for [remote "origin"] $ cat .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [user] name = Avery email = avery@example.com [remote "origin"] url = ../remote/ fetch = +refs/heads/:refs/remotes/origin/ </code></pre> Remotes are configured in the local repository config file. The url</code> option is the location of the remote - in this case, a relative path to the remote on our local filesystem. More formats are supported, including https://...</code> and ssh://...</code>.^{8</a>} Cloning is the normal way to get a copy of an existing repository. The cloned URL will automatically be added as the origin</code> remote, but more remotes can be added and existing ones adjusted with $ git remote set-url</code>. Simply copying someone's .git</code> directory would also work, but you would end up copying data which you probably don't want, such as their personal configuration, stashes, local branches, and unreachable objects. Fetching</h3> In order to collaborate, several people will use the same remote in their local repositories. Local repositories download the refs and objects stored in remote repositories through $ git fetch</code>, which is known as fetching: $ git fetch From ../remote [new branch] master -> origin/master </code></pre> Fetching is controlled by the fetch option for the remote: +refs/heads/:refs/remotes/origin/</code> - this means "download all refs under refs/heads/</code> in the remote repository to the local repository, and store them locally under refs/remotes/origin/</code>". All objects that are reachable from those refs and not stored locally are also downloaded and saved in the local repository's object store. With fetching, we have a mechanism for synchronising from a remote repository to a local one, and have new refs for keeping track of heads from remotes. These are called remote-tracking branches: $ git log --oneline --graph --all 10498bd (HEAD -> master, origin/master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Logs now show the origin/master</code> head alongside master</code> for commit 4, showing that these heads have been synchronised since the last fetch. Note how remotes/</code> can be (and usually is) omitted from the head name. Pulling</h3> Whilst fetching will simply download the remote's refs and new objects, you may also want to incorporate changes others have made into your local branches. This is done with $ git pull</code>, and is called pulling. Pulling will simply fetch, then merge your current head with its counterpart in the remote. The counterpart is not set by default - set it with $ git branch --set-upstream-to=<remote>/<branch></code>. For example: $ git branch --set-upstream-to=origin/master Branch 'master' set up to track remote branch 'master' from 'origin'. </code></pre> This will add a branch</code> section in .git/config</code>: [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [user] name = Avery email = avery@example.com [remote "origin"] url = ../remote/ fetch = +refs/heads/:refs/remotes/origin/ [branch "master"] remote = origin merge = refs/heads/master </code></pre> If we now attempt pulling: $ git pull Already up to date. </code></pre> As the origin and local heads are equal, no merge is necessary. If there were a difference, a merge commit would be created or your local head would be fast-forwarded. In the former scenario, this would create a new commit with the sole purpose of incorporating the changes others have introduced, which isn't ideal. Instead, if you already have new commits on your local branch, you can rebase them onto the origin's head with $ git pull --rebase</code>. If you prefer this over the default behaviour, you may be interested in the config options pull.rebase</code></a> and branch.autoSetupRebase</code></a>. Don't rewrite shared history</h3> Avoid changing history that already exists on a remote. Essentially, this means not changing a history that currently contains an existing remote's head into one which no longer contains that head. For example: $ git log --oneline --graph --all 10498bd (HEAD -> master, origin/master) Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Currently the remote head origin/master</code> exists in the history of our HEAD</code>, pointing the same tip commit. If we were to amend the tip commit, this would no longer be the case: $ rm file1.txt $ git commit --all --amend [master 33cda34] Fast-forward commit Date: Fri Jan 1 00:00:00 2021 +0000 1 file changed, 1 deletion(-) delete mode 100644 file1.txt $ git log --oneline --graph --all 33cda34 (HEAD -> master) Fast-forward commit | 10498bd (origin/master) Fast-forward commit |/ 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ 173bb18 (tag: first) Add file1.txt </code></pre> Now in branch master</code>, 10498bd</code> has been replaced with 33cda34</code>, so the local and remote branches no longer share a common ancestry. We can still force the remote to update its head to match our new local head with $ git push --force</code>, but there's the possibility that someone else has already sent commits to the remote, and these commits will be lost because our local repository doesn't have them in its history. This can also cause trouble for everyone working locally on the branch with the old history. When they try to pull your changes, they could encounter conflicts on commits they didn't create, and be forced into a confusing merge or rebase as they try to apply the old history on top of the new history of the remote. With common branches like master</code>, people will be regularly pulling and applying new changes on top of the remote's version of that branch. It's particularly important to maintain the existing history of these branches to avoid inconveniencing its users. As such, it's better to create new commits rather than trying to rewrite the existing ones. For example, if you want to undo a commit that hasn't been pushed yet, you can drop said commit with a rebase or reset. If that commit has already been pushed, use $ git revert</code></a> to create a new commit that undoes the changes in another. Thankfully, remotes will prevent you from rewriting branch history by default. However, this isn't an absolute rule, and you can force remotes to accept the rewritten history in cases where it makes sense. For example, if you're working on temporary feature branches, it may be acceptable to first rebase them before merging them into a main branch. Your team should set rules on when shared branch history may be rewritten, and shared repositories can be configured to block forced rewrites to enforce those rules. If you find yourself having accidentally rewritten shared history, try cancelling your current merge or rebase, or use the reflog</a> to (mixed) reset</a> back to the commit before you rewrote history: $ git reset 10498bd HEAD is now at 10498bd Fast-forward commit # Or reset --hard (if you want to also reset the working tree and index!) $ git reset --hard 10498bd </code></pre> You may then have to cherry-pick</a> any new commits created on top of the rewritten history. Pushing</h3> Finally, you can push (i.e. upload) changes to a remote with $ git push</code>. $ git commit --allow-empty --message "Pushed commit" [master 66d24b9] Pushed commit $ git push Enumerating objects: 1, done. Counting objects: 100% (1/1), done. Writing objects: 100% (1/1), 184 bytes | 184.00 KiB/s, done. Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 To ../remote 10498bd..66d24b9 master -> master $ git log --oneline --graph --all 66d24b9 (HEAD -> master, origin/master) Pushed commit 10498bd Fast-forward commit 031b880 Merge my-branch into master |\ | 630d4e3 Commit on my-branch | 0538d0d Commit on master |/ * 173bb18 (tag: first) Add file1.txt </code></pre> This uploads the head of the current branch and any associated objects, then updates the remote head in our local repository. If your local history isn't up to date (i.e., someone else has pushed commits since your last pull), you'll be met with an error like this: To ../remote/ ! [rejected] master -> master (non-fast-forward) error: failed to push some refs to '../remote/' hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Integrate the remote changes (e.g. hint: 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. </code></pre> In this case you'll need to perform a pull. You'll get the same error if you rewrote shared history</a> and caused the remote head to be lost in your local history. If you're sure you want to push these changes anyway, you can use $ git push --force-with-lease</code> to ignore the error and force the remote to update and match your local history. This option checks that the heads in the remote match your remote-tracking branches, so you won't accidentally lose commits that have been made since your last fetch. However, this option is not infallible; it would fail to protect you if another process is updating your tracking branches in the background.9</a> Similarly, the --force</code> option also ignores the error, but doesn't perform any checks on the remote heads, so it should be used with even more caution. As mentioned previously, remotes can configure whether or not forced pushes are allowed on individual branches. Extra resources</h2> Listed below are extensions and additional functionality and you may find useful if this article has held your interest: .gitignore</code></a> files let you configure which files shouldn't be tracked. </li> $ git bisect</code></a> lets you efficiently search history for changes that introduced issues. </li> $ git worktree</code></a> lets you have multiple working trees that share a single repository. </li> $ git sparse-checkout</code></a> lets you create a working tree that contains a subset of files in a repository. </li> Submodules</a> let you to maintain repositories as a subdirectory of another repository. </li> git-svn</a> is a bridge for using Git with an SVN repository. </li> Git Large File Storage</a> and git-annex</a> are extensions which let you use Git to track large files without bloating local repositories with their full histories. </li> </ul> If you'd like to learn more about branching models, take a look at git-flow</a> and GitHub flow</a>. If you prefer GUIs, Git has a built-in interface which you can launch with $ git gui</code>. The Git website maintains a list of third-party GUI clients here</a>. If you're an Emacs or Vim user, I'd recommend checking out Magit</a> or Fugitive</a> respectively. References</h2> ^1https://missing.csail.mit.edu/2020/version-control</a> </div> ^{2 Git For Ages 4 And Up by Micheal Schwern @ Linux.conf.au </div>} YouTube</a>, Wayback Machine</a></li> </ol> ^3Git From the Bits Up by Tim Berglund @ JAXconf 2013. YouTube</a>, Wayback Machine</a> </div> ^4https://git-scm.com/docs/gitglossary</a> </div> ^{5</div>}https://github.com/bk2204/git/blob/transition-stage-4/Documentation/technical/hash-function-transition.txt</a> ^{6</div>}https://git-scm.com/docs/git-diff-tree#_diff_format_for_merges</a> ^{7</div>}https://git-scm.com/docs/git-reflog#Documentation/git-reflog.txt---expirelttimegt</a> ^8All supported remote URL formats are listed in https://git-scm.com/docs/git-fetch#_git_urls</a> </div> ^{9</div>}https://git-scm.com/docs/git-push#Documentation/git-push.txt---force-with-leaseltrefnamegt</a> Do an internship Unknown — Mon, 22 Feb 2021 00:00:00 +0000 Should you find yourself in further education, or are otherwise looking to get started in an industry, I'd strongly recommend taking part in an internship. I personally found mine incredibly helpful as an undergraduate student. It boosted my confidence, gave me a better idea of what direction I wanted to take my career in the future, and made it easier to get started once I left university. My experience was working as a software developer in a private corporation. Even if you're not looking at the same field or the private sector, I'd still recommend an internship. While I might mention a few things that are specific to my field, I hope this article helps anyone who's entering a new field or heading into their first real job. A break from academia</h2> If you're young, chances are you've spent most of your last decade in full-time education, continuously working from exam to exam. At several points throughout my student life I've found it difficult to find the motivation to study and felt drained from stressing about my performance. Taking a break from the education system helped me find the motivation to finish my studies, gained me some perspective on just how different life is in the professional world, and how much I preferred it. I felt much more motivated and engaged during my placement than I had been for years as a student. An important part of professional life is setting strong boundaries between working and non-working hours. Having set hours means no guilt over not working on those assignments or revising or doing some last minute adjustments to coursework. Outside of those set hours, work isn't on my mind. In any respectable organisation, you'll be able to determine what hours you work and, unless you give consent to the contrary, you won't be expected to concern yourself with work outside those hours. Another motivator is receiving compensation. Knowing that you're being paid for your work makes it much easier to concentrate and get your work day done. Additionally, having your own source of income after relying on others gives you additional independence and agency over your life. Earning some savings during your internship can provide an additional cushion to get you started once you move into a permanent position. Developing yourself</h2> While you've likely learned a lot during your studies, it takes an additional set of skills to apply them effectively when working in a team. If you're lucky you'll have worked on a group project and have some firsthand experience. It takes time to learn how to effectively communicate, plan & organise (team) work, document information for others, and use other tools of your trade. In software engineering specifically, that means learning to use version control software, issue tracking systems, writing technical documentation for your fellow engineers, and non-technical documentation for others. In each of these there are additional considerations that arise when working in a large team, and when joining an existing project. Learning is a big part of any internship (or any job, for that matter), but getting to apply your existing knowledge and skills is very satisfying. While you'll likely experience some imposter syndrome at the start, you'll soon accept that your fears over your own competence were unfounded, and that you deserve the internship position that was offered to you. That'll also help you realise that, yes, you actually are ready for the "real world". At the end of the experience there's a good chance you'll be offered a permanent position, which will give you more options when leaving academia and ease that transition. Getting used to full-time work</h2> The first issue I struggled with was imposter syndrome: feelings of inadequacy and concerns over how I was performing. My first project was initially a solo one, which I kicked off. While I had a coworker beside me to ask any questions to, my lack of experience made it difficult to gauge my rate of progress and what was expected. After a couple of weeks my manager let me know that he was happy with how things were going. This alleviated my concerns for a while, but the occasional doubts crept in during the first few months. Continued positive feedback on my performance eventually convinced me there was no need to worry, and that I was doing well. I realised that feedback typically comes infrequently, and lack of it usually means things are fine; any manager will let you know when they have concerns (and if any managers are reading this: remember that your interns need more feedback than experienced professionals!). I also could have simply asked for feedback, especially when I was unsure of how I was doing. It's also always a good idea to ask for feedback during your reviews so you know how you can improve. Burning out was also a concern. On quite a few days in my first weeks, I finished work later than I needed to due to being too engrossed in my work, or trying to wrap things up before I left for the day. Over time I realised this was causing me to feel a lot more tired after work than I would be otherwise. It takes some time to find the proper balance while working in a full-time job. Work can be both physically and mentally draining, and overworking yourself will simply result in both diminishing returns and unhappiness. It's okay to give yourself breaks and take your mind off your work for a little while, especially when you feel like you're not getting anywhere. In creative jobs, breaks give you a chance to reset, figure out different ways of approaching your work, and notice mistakes and missed avenues. It's not sustainable to apply your full effort at all times. When you have some down-time or feel like you need a break from your main task, swap to a less draining one like writing documentation or doing some related reading. Keep in mind</h2> Make sure that you're being appropriately compensated for your work. Put some time into researching the average salaries for the position you're looking at, both internship and permanent. Make sure that the compensation fully covers your living expenses and leaves a comfortable amount to spare. I'd strongly recommend against taking an unpaid position, especially a long-term one. Sadly, private companies tend to offer salaries that are much lower than what they'd offer a permanent candidate. I'd personally suggest being wary of internships that offer low salaries, especially if you're hoping for an offer for a permanent position after it is completed. A company that starts you off with a low salary is likely to keep offering you low salaries if you continue with them. If you're looking for a year-long student placement while you're at university, I'd suggest researching if you can suspend your studies during it. I did my placement as part of my degree, during which I was charged 20% of my normal tuition fee despite receiving almost no university work or contact hours. An internship should give you experience that you can directly apply in the field you're looking to work in after the internship. Be sure to check and ask about the responsibilities of the positions you're looking at, ensuring that they would be useful to your career - try to find things you could to mention to prospective employers in the future. If find yourself regularly being tasked with menial work unrelated to your field, I'd suggest talking to your manager about it, and potentially leaving your internship if things don't improve. Finally, make sure to hydrate yourself throughout the day, and in desk jobs, get up every now and then to exercise your legs! After your internship</h2> Getting some first-hand experience in a job tells you another thing: what your preferences are. A good internship will expose you to a variety of aspects of your field, e.g. frontend/backend website/application development, different technologies, different roles within a team. You'll also have found out how you like to work. Do you dislike wearing a suit to work? How do you feel about open offices? Would you prefer flexitime or remote working over a traditional 9-5 in-office work week? This helps inform your choices after your internship, i.e., what kind of roles you look for, and in which kinds of organisations. Make sure to ask about the things that are important to you in future interviews. In the end, a good internship offers some invaluable benefits: work experience, a positive reference, and a foot in the door. Even if you're not offered a full time position or choose not to accept it, having your internship on your résumé will show employers that you've proven your abilities in the past. Diagnosing issues with tcpreplay & tcprewrite Unknown — Wed, 23 Sep 2020 00:00:00 +0000 Tcpreplay</a> lets you take previously recorded network traffic (e.g. with tcpdump</a>) and send it out again, keeping packets the exactly same or altering them slightly. This can be handy if, for example, you're testing an application that receives data over a network. Recorded packets include the original metadata, including the source and destination addresses, so if you want those recordings to instead transfer data between completely different computers, you'll have to adjust them beforehand. tcprewrite</code></a> is another tool designed to do exactly that. This article goes in to what to look out for when you're having problems using these. Step 1: Check that the target destination can receive data</h1> Use tools like netcat</a> and Wireshark</a> to try sending packets from the source computer to the destination computer. For example, $ nc 192.168.0.20</code> will send a TCP packet to that address, and $ nc -u 192.168.0.20</code> will send a UDP packet for every line you enter. Wireshark and similar tools can be used to detect these packets on the receiving end. If these packets are detected as expected, but the packets from tcpreplay</code> are not, your issues likely lie with tcpreplay</code> rather than your network. Run tcpdump -e</code> (specifying the correct interface with -i</code>) while manually sending packets with netcat</code>, identify the lines caused by that traffic, then run tcpreplay</code> and compare the packets. Another issue can be additional unwanted metadata in packets, including 802.1q VLAN tag information. Try stripping it from the packet capture using tcprewrite</code>'s --enet-vlan=del</code> option. Finally, if you're expecting the recording to include multicast UDP traffic, make sure the destination application registers itself as part of the appropriate UDP multicast group, which is the destination address specified in multicast packets. For some reason, when I was using it to diagnose issues, Wireshark did not display multicast traffic, causing me to believe tcpreplay</code> was at fault. However, an application I made to detect multicast packets successfully picked them up once the socket was configured to join multicast groups. Step 2: Correct MTU size issues</h1> When using high replay multipliers (with the -x</code> option), I ran into errors indicating packets had exceeded the MTU size. If increasing the MTU of your network devices isn't an option, I'd recommend trying tcpreplay</code>'s --mtu</code> and --mtu-trunc</code> options. Step 3: Rewrite addresses correctly</h1> An obvious problem can be that the source/destination addresses are incorrect within the packet capture. Use tcprewrite</code></a> to correct those, particularly looking at the --pnat</code>, --srcipmap</code>, and --dstipmap</code> for rewriting IP addresses, and --enet-smac</code> and --enet-dmac</code> for correcting MAC addresses. Step 4: Correct packet checksums</h1> Packets with incorrect checksums may be dropped by the destination computer. Wireshark can indicate these packets with the advanced options ip.check_checksum</code> and protocol variations such as udp.check_checksum</code> and tcp.check_checksum</code>. Recorded packets with incorrect checksums will then be highlighted in black/red. If you run into this issue, add another (maybe separate) pass of tcprewrite</code> using the --fixcsum</code> option. Good design patterns: the command palette Unknown — Sun, 06 Sep 2020 00:00:00 +0000 Concept</h2> I first encountered the name "Command Palette" from the Sublime Text</a> editor. The idea is simple: your application opens a menu with a textual list of possible actions that your application can perform. Add a searchbar on top that's active immediately, so that one can start typing to filter down the options until the desired action is found. These options are gathered from the application's menu bar, as well as from submenus within those menu options. As you search, the matching text in each option can be highlighted, drawing your attention to the region of the option you're likely to be interested in. Additional information about each option can be displayed, including associated keyboard shortcuts, or the path to the option in the menu bar. The search term can not only filter based on the name of the option, but also on hidden things a user might know and search for, e.g. the description of an action, or a command to run (cmd</code> for Command Prompt). Command palettes are most powerful when they can be opened through a keyboard shortcut, as that enables simple keyboard-driven interaction with the application. Other editors took inspiration from this feature of Sublime Text, including Visual Studio Code</a> and Atom</a>, so clearly the concept is rather popular and functional. Benefits</h2> The primary and universal benefit is discoverability: a user can use (limited) natural language to describe what they want to do, and the application presents them with the closest-matching results. This lets users quickly access and perform what they need, reducing their frustration, particularly those who are new to your application. The worst thing you can do is force new users to dig through several menus from a menu bar or ribbon, hoping that your developers and designers have organised things in a sensible manner1</a>. Users that prefer the menu bar interface can be taught where to find the actions they searched for, e.g. by showing a sequence (File → Submenu → Action), or by expanding the menu to show the item, as macOS does: Screenshot source</a> Power users like me, who dislike being forced to use a mouse, will greatly appreciate having an effective keyboard-driven UI. The command palette can also show keyboard shortcuts associated with each action, so one can easily learn even faster ways of performing regular actions. It's also a significant improvement over the traditional menu bar when it comes to keyboard control. The process for a menu bar involves hitting Alt , looking for an underlined letter or using the arrow keys, expanding a submenu, then repeating the navigation process until ones reaches the desired action. Conversely, with the palette, one can bring it up with a keyboard shortcut, type one or two words, and hit enter. Admittedly, the former also allows muscle memory to be built up, but the sequence of key presses is non-obvious and specific to each action, and hence takes longer to learn. A word or two is much easier to remember, compared to a seemingly-random sequence of letters which often has no particular relation to the menu items I'm activating. Doing it right</h2> 1. Show additional contextual information</h3> macOS takes the command palette concept one step further by building the feature into the OS: all apps that use the standard menu bar will have automatically have a search bar under the Help menu, with the hotkey ⌘ Command ⇧ Shift / . This is a feature I loved while using macOS. No matter how badly designed the application, this feature would save me the hassle of using a pointer and keep my hands on the keyboard most of the time. Such a feature is sorely needed in many Windows applications, many of which I've found to be heavily mouse-driven, sometimes even forgoing the traditional menu bar, and thus removing an avenue for keyboard navigation. The Windows 10 stock apps like the calender and email client are particularly guilty of this. Apparently we have their new "Design System", Fluent</a>, to blame. An alternative to visually showing the location of options in the menu would be to show a textual representation of its location. Showing the keyboard shortcuts of each option helps the user speed up in the future by skipping the command palette entirely. 2. Avoid inconsistent or astonishing search results</h3> The command palette is at its best when users can reliably include it in their workflows within your application. The same query should return the same result, no matter how quickly it was typed in (within reason: new features and context are bound to change some results). An example of this done poorly is the Windows 10 search bar: one can use it as an application launcher by searching for the name of the desired application or system menu and simply hitting enter. But there are problems that occasionally cause frustration, such as the applications you want randomly not appearing or internet search results being displayed instead of locally-installed applications. In order to fix this, the search menu should: Wait until the local search is complete before executing the first result</li> Prioritise local results over internet results (and give an option to disable internet searches)</li> </ul> Some command palettes such as Rofi</a> and Wofi</a> sort matching options based on how frequently you've chosen those options. I personally find this more detrimental than helpful, as over time the order of options will change, affecting the results of quick searches I've memorised. My usage pattern involves typing the minimal amount to get my desired option as the top result, then hitting enter - if there are two similar options I regularly use, I've found they can clash and make it difficult to learn the quickest way to pick one or the other. For users who rely on learning the bare minimum they need to type to get the desired result without inspecting the search results, I'd recommend making such behaviour optional. 3. Sort results sensibly</h3> The sorting issues I encounter when using Wofi would be less impactful if the result sorting used additional heuristics than simply how often the user has selected it. Here are some additional things to consider and prioritise: Did search term match the shown text or hidden text? The shown text should be prioritised, as that is what the user is more likely to search for if they know exactly what they're looking for. </li> Did search term match the beginning of (or entire) word(s)? People are much more likely to start typing words from their beginning, so prioritise such matches. Similarly, whole word matches should be prioritised over partial word matches. </li> </ul> 4. Show as many options as available</h3> The worst crime against discoverability is simply withholding options from your users in a command palette. The command palette should provide you with as many options as are available through traditional means, e.g. the menu bar or search bars within the application. Sub-menu items can also be expanded into the palette, perhaps with a prefix to indicate what menu they come from. To take this even further, if necessary, palette options could open another palette with sub-options. This is another issue I take with Windows 10's search bar: it intentionally hides things like old system menus in favour of pushing their often less-useful modern counterparts. This is more understandable if the new option is a complete replacement, but otherwise it's an unnecessary limitation of the search function. Summary</h2> The command palette is a simple design pattern, and so it's easy to learn. It lets your users find what they need more easily and quickly than other traditional menu designs, teaches them what your application can do, and how to do it. This design is one I greatly appreciate having in any application, and I'd love to see it used more widely in the future. ^1For an example of what not to do, see Martin Keary's critique of Sibelius' UI</a> </div> Domain name hacks Unknown — Tue, 04 Aug 2020 00:00:00 +0000 An increasingly common trend is the use of domain hacks, wherein one uses TLDs in an unintended or unconventional way. I've noticed this recently in the primary instance of Invidious</a>, invidio.us</a>. While incorporating the TLD into the domain name like this might be more visually appealing and perhaps elegant, there are several downsides to doing so in my view. Each time I type that URL I find myself having to backtrack two characters to insert the period for the TLD. I perceive it as a single word, hence my typing habits oblige me to type the word in its entirety without pausing. If I attempt to insert the period before completing the word, it breaks my flow and I can't easily work out where the period should go. Now imagine you're telling someone to visit one of these domains - how do you pronounce it? "Invidio, dot, us"? Or "Invidious with a dot between the o and u"? Imagine dictating such a domain to a less technologically savvy individual who is only used to hearing "dot com" and their regional equivalent. Of course, they may still be able use a search engine to find what they're looking for unless they're typing in a shortened URL (the horror). All of these are unnecessary inconveniences introduced just so your domain name looks a bit prettier. Not using appropriate TLD also removes information normally contained within a domain name: the type of entity the website is for. .com</code> for company, .net</code> for networks, and regional variations of each. TLDs have a semantic purpose - a purpose which is discarded if they're used in a hacky name. Of course, this website is itself guilty of using a TLD in an unintended manner - .me is for Montenegro, but here it instead represents a personal website. This hack is so commonly used to the point that this alternate meaning for the TLD is widely accepted, and even touted by the registrars</a>. Thus, I thought it was an acceptable choice for my own personal website. Ideally there would be an TLD that better applied to a generic personal website. There are some other TLDs I considered, before I eventually settled on .me as being my favourite option: .pro</code>: My website doesn't contain exclusively information relating to my professional career; it's also for personal purposes</li> .info</code> & .site</code>: Too generic - all websites are "sites", and virtually all of them contain information. This site also contains information that isn't specifically related to me</li> .blog</code>: My website isn't exclusively a blog</li> Regional TLDs: I didn't want to pigeonhole myself into a particular location in case I relocate at any point in my life. Further, where I live isn't a significant part of my identity</li> Other new TLDs: Could confuse people who didn't even know they existed, and/or don't know how to type them</li> </ul> Unfortunately transferring to a new domain can be quite a pain, so for the foreseeable future I'm stuck on williamvds.me, even if a more appropriate TLD appears. Software I use Unknown — Fri, 22 May 2020 00:00:00 +0000 Personal computers</h2> You can find most of my configuration files here</a> . Linux</h3> My computers run a distribution of GNU/Linux, specifically Arch Linux</a>. It's the first distribution I installed and it's served me well, so I haven't had any reason to try another. Here's some things I like about it: The Arch User Repository (AUR)</a> lets anyone provide the build scripts for creating packages that aren't in the main repositories. This makes it far easier to keep all your installed software in packages.</li> Rolling releases: packages are updated shortly after upstream releases, meaning I can quickly get my hands on new features and bug fixes. It's been a surprisingly stable experience - the only time I ran into an issue after updating was while I was using the testing repositories.</li> </ul> Sway</a></h3> A tiling window manager that uses the new Wayland protocol. It's mostly compatible with the more well-known i3 window manager</a>. I started using tiling window managers about a year after I switched to GNU/Linux, and it's tough to go back to a traditional one. Here's some things I like about them: Configuration files are human-readable, hence easy to edit and store in version control</li> Keyboard shortcuts are available for all actions I regularly perform (switching and organising windows, etc.)</li> Assigning windows to virtual desktops and automatic layouts means almost no fussing about with window layouts, and I can easily get to the app I need</li> They're relatively lightweight but heavily customisable (see the "ricing" subculture)</li> </ul> While I was on macOS at work, I tried using chunkwm</a> to recreate the tiling window manager experience. However, getting the window manager set up was a frustrating task, and I never got it working reliably and how I wanted it. Minor inconveniences like compulsory desktop transition animations and key-binding limitations quickly add up to make a miserable experience. chunkwm's author has more recently released a new window manager: yabai</a>. Hopefully that provides a better experience, but for now, I just try to avoid macOS as much as possible. Firefox</a></h3> I've been using Firefox since a bit before the 'Quantum' update. I'm happy to say it's now a snappy and stable experience. Most settings can be controlled in the human-readable user.js</a>, and the browser can be themed using the userChrome CSS files</a>. Some extensions I recommend: uBlock Origin</a>: The only browser ad-block extension you should use. I use uBlock Origin's dynamic filtering feature</a> to block third-party resources, so pages load quickly by default, but some sites require configuration to load properly.</li> Decentraleyes</a>: Stores a local copy of commonly used JavaScript libraries so you're not fetching them from Google or wherever.</li> Better Image Viewer</a>: Improves image viewing with stuff like scrolling to zoom and dragging to pan.</li> Dark Background Light Text</a>: Adjusts the colours of webpages to make a dark mode for every website.</li> Multi-Account Containers</a>: Segregates your cookies and stuff into categories. Gives a privacy boost and is convenient when managing multiple accounts on a single website.</li> Hover Zoom+</a>: Hovering over links and thumbnails will show a maximised image/album/video.</li> SponsorBlock</a>: Skip sponsorship segments in YouTube videos.</li> Tab Center Reborn</a>: Shows your tabs in the sidebar. Combine it with userChrome</code> styles to make the most of your vertical monitor space.</li> Privacy Redirect</a>: Redirect YouTube, Twitter, and others to their lighter alternative frontends. Lets you avoid YouTube's region & age restrictions and Twitter's login barriers.</li> Tridactyl</a>: Vim-like key bindings in your browser.</li> PassFF</a>: A browser plugin for my password manager, pass</a>.</li> </ul> Things to improve: Replace Tab Center with Tree Style Tab</a> to improve tab organisation.</li> </ul> Neovim</a></h3> Vim's yet another piece of software that's difficult to give up once you've gotten into it. I picked it up when I was stuck in a university computer lab with nothing else to do and typed $ vimtutor</code>. The learning curve is infamously harsh, but keep at it and you'll learn a text editing model that you can keep using for the rest of your life, and writing on computers will never be as mundane again. I chose to use the Neovim fork because of its refactoring efforts and its ability to allow plugins to work asynchronously (though original Vim now also supports this). With a few plugins and tools Vim can also become a simple IDE, thanks to the Language Server Protocol</a>. Enabling code completion, refactoring, and searching is now trivial for programming languages that have an existing tool that implements the protocol. If all else fails, most popular IDEs have an add-on that provide a Vim-lite editing experience so you can keep all the useful features your IDE provides you. Newsboat</a></h3> A terminal RSS reader which can synchronise with my Nextcloud service. Heavily customisable, just how I like it. aerc</a></h3> A simple terminal mail client. I swapped to aerc from NeoMutt</a> as I preferred the simpler interface aerc goes for. aerc handles IMAP and SMTP itself so there's less configuration involved to get it set up. It's got a fair number of features that make it easy to handle Git patch requests and the like within the client. While I haven't yet used these features it sounds useful for anyone involved in open source software development. I pair it with OfflineIMAP</a> so I can keep a copy of my emails locally on my computers. Things to improve: Proper notifications on new emails</li> Adjust appearance?</li> </ul> Bash</a></h3> It's the standard GNU/Linux shell, so I prefer to use it whenever possible. With a few configuration adjustments I think it stands up well to the competition (I've also tried zsh</a> and fish</a>). Using it as a scripting language is a different story. Lots of subtle behaviour and its limited feature set make it a hassle to create simple and correct scripts. I'd recommend using another scripting language for more complex scripts, else use ShellCheck</a> religiously. I use an adjusted Agnoster</a> theme for a cleaner and more detailed command prompt. Things to improve: Find something equivalent to fish's abbreviations</a>: they'd make command completion for aliases much less of a hassle.</li> </ul> tmux</a></h3> A window manager for your terminal. I use tmux in much the same way as I do sway, with sessions started automatically and programs launched in them. Ideally I'd just use my actual window manager, but being able to connect remotely and still have all my terminal sessions open is a killer feature I can't easily recreate with sway. I've set up a few two-key bindings that are similar to the ones I use in sway; I use the Super</code> key modifier in sway, and Alt</code> in tmux. Things to improve: Show SSH connection and Git status in tmux bar (see tmux-gitbar</a>) instead of the Bash command prompt.</li> </ul> Self-hosting</h2> This website</h3> Being a lightweight static website, it's no hassle to set up and host it from my own server. I originally created it with Vue.js</a>, as I wanted a dynamic résumé I could adjust over time and tailor for particular job applications. While I pushed this Vue.js version to my live server, I later realised the same could be achieved with a static site generator. I eventually settled on Zola</a> as it seems relatively lightweight, has a good selection of features, and doesn't use JavaScript. It can easily be maintained in a Git repository, which allows me to push changes both to the origin repository (wherever that may be) and my server easily. With a Git hook I can have my server regenerate the site whenever I push changes, automating the deployment process. You can find the source for my website here</a> . Nextcloud</a></h3> A great "cloud" suite that provides file and calendar syncing. Once I had this set up I was kicking myself for not having done so sooner. Having all my files automatically available on all my devices is a wonderful experience and Nextcloud proves you don't need commercial solutions to do it. The automatic syncing has been so hassle-free I've caught myself relying on it in scenarios where I should be using a Git repository. Note to self: stop that. Things to improve: Stop relying on it as a backup solution, and instead perform full backups kept in cold storage.</li> While contact syncing is available it doesn't seem to function properly with Android - currently it's only making regular .vcf</code> backups.</li> Make a dark theme for the web UI.</li> Create a good terminal client: nextcloudcmd</code> is bundled with the full Nextcloud QT5 client, which is rather hefty and depends on QT5's web engine.</li> </ul> Taskwarrior server</a></h3> Taskwarrior is a powerful task tracking and time-keeping tool which I severely underutilise. Its appeal is how it can automatically prioritise tasks for me: I can input tasks and have a prioritised list always telling me the next thing I should be doing. Used properly, I see it being a great force for productivity and applying order when necessary. I've set up a synchronisation server so my tasks can be shared between all my devices and I can keep up with them wherever I am. Things to improve: The Android app</a>: I forked the original with the aim of refactoring, modernising, and adding new features to it. The current state is an improvement in some aspects, but some functionality is broken. Also, sync messages regularly fail and produce annoying error notifications.</li> Work out how to use recurring tasks properly.</li> Try to use it more often</li> </ul> Firefox Sync Server</a></h2> Allows you to store your synced Firefox data on your own server instead of Mozilla's. Raspberry Pi</h2> Pi-Hole</a></h3> Blocks advertising for all devices in your LAN. What else do you need to know? Things to improve: Configure the WiFi hub so all devices actually pick up the Pi as the DNS and DHCP server.</li> </ul> Android smartphone</h2> LineageOS</a></h3> An open-source community-run modified distribution of the Android Open Source Project</a>. Supports a wide range of Android devices, including ones that aren't being updated by the manufacturer. Even if the manufacturer doesn't support it, they'll often support the latest versions of Android for your device. Their changes from AOSP are rather minor, and add nice new features rather than adding bloat like many custom versions of Android tend to do. LineageOS still relies on proprietary drivers and such, so it's not entirely open-source, but probably the closest you're going to get for your device. Installing Google's proprietary apps and services</a> is optional, but expect some apps (particularly proprietary ones) to misbehave with regards to push notifications. The microG</a> project provides an open-source re-implementation of Google's things, but requires modified LineageOS builds in order to pretend to be those apps. microG hasn't released builds for the latest version of Android yet, and I'd rather not rely on Google services at all, so I've not yet tried it. I'm content to have a mostly functional mobile device that is as open-source as possible rather than rely on proprietary software and services, thus, the only option from the iOS/Android duopoly is the latter. In the future I may take a look at postmarketOS</a> for a true open-source operating system for my phone. AdAway</a></h3> Provides system-wide adblocking through hosts files. Sadly it does require root access</a>. AntennaPod</a></h3> For listening to podcasts, and has about all the features you could need. AnySoftKeyBoard</a></h3> Highly customisable and feature-rich, including the wonderful swipe-typing. A bit temperamental with suggestions and swiping, but mostly works. Sadly the settings UI is a bit of a mess and it's difficult to find what you're looking for. Aurora Store</a></h3> An open-source client for using Google's Play Store. Handy when you really need a proprietary app, or when a FOSS app is only distributed through the Play Store (looking at you, Signal). Simple Calendar</a></h3> A simple calendar application. Looks and works better than the stock LineageOS app. DAVx⁵</a></h3> A CalDAV/CardDAV/WebCal client, useful for hooking up your phone's calendar and contacts to Nextcloud</a>. F-Droid</a></h3> The FOSS app repository for Android. You can download and update the vast majority of what you need from it. Firefox Preview</a></h3> A new preview version of Firefox for Android. The UI and performance is a bit better than the current version, and it's being actively worked on. I'll admit the UX still doesn't touch Google Chrome's, but I hope that'll soon change. Magisk</a></h3> Grants you root access to your device. NewPipe</a></h3> An open-source client for YouTube, SoundCloud, and some FOSS media services. Provides features for free that these platforms charge for, including being able to download videos, listen or view them in the background, and remove adverts. Nextcloud</a></h3> I use the official Nextcloud and Nextcloud news apps. The unofficial notes app</a> provides a better experience when editing notes, so I prefer it over the normal Nextcloud app. OsmAnd</a></h3> An OpenStreetMap</a> client complete with offline maps, offline route calculation and navigation, and customisation. Public transport routing is almost working, but in London, it often takes a long time a route or doesn't load at all. In the rare case where I need public transport routing, I just the service provided by the public transportation organisation. Password Store</a></h3> An Android client for pass</a>, which I use as my password manager. Requires OpenKeychain</a> for PGP key functionality. Things I want to do</h2> Chromecast</h3> Google's Chromecast is a nifty little device that lets you stream media to a device. After cutting Google out of my Android device, my Chromecast device is now practically useless: support for it seems to be provided by the Google Play Framework</a>, and none of the Android apps that I use implement support for the Cast API themselves. While projects like pychromecast</a> provide an API for controlling Chromecast devices from a desktop, it's hardly the best experience. I created a Python program for queuing YouTube videos with pychromecast which serves my basic needs, but things like manipulating the play queue isn't possible. On Android, it seems like there's a similar effort for an open-source implementation of the Cast client API in chromecast-java-api-v2</a>. I hope this project finds itself being used by other open-source Android applications, as I'd like to have Cast protocol support again. Ideally we'd have an open-source implementation of the Cast server so one could create their own Chromecast devices, e.g. using micro-computers like the Raspberry Pi. Something like Mopidy</a> combined with Snapcast</a> and/or Home Assistant</a> could provide an alternative. Home Assistant</a></h3> Home Assistant is a widely lauded home automation suite, which I found myself setting up on a Raspberry Pi. That part was simple enough. Once I had it running I quickly realised I didn't have anything to do with it. I found myself disappointed with its Chromecast support, as the interface only shows the currently playing media and a play/pause button. Perhaps if I get some "smart" home devices I'll find a better use for it. Some ideas: Listing schedule & events</li> Coloured/dimmable lights Adjust light temperature and intensity at a regular schedule to maintain a regular circadian rhythm and provide light therapy during winter.</li> </ul> </li> Temperature & air quality sensors Warn me when CO₂ levels get too high</li> </ul> </li> Automated lighting system with Bluetooth presence detection</li> </ul> Combined with a voice interface or one of those magic mirrors</a>, I can see Home Assistant being a great tool for automation and personal management. Media server with Kodi</h3> Ideally I'd have a small computer with mass storage that could act as a server within my LAN, and could store media that I want to keep at hand but not necessarily on my PC. Combined with Kodi</a>, such a computer could provide a great home-entertainment system. Automate personal computer setup</h3> My user configuration files ("dotfiles") are stored in a Git repository</a> , which includes an installation script. This isn't ideal for the purposes of quickly reproducing my setup on a new computer. A configuration management tool like Chef</a> or Ansible</a> would be better suited for this task. A better terminal client for Signal</h3> I use Signal</a> for my instant messaging. Sadly, the desktop application is an Electron application and hence regularly gobbles up a gigabyte of my system's memory for no good reason. signal-cli</a> and scli</a> are almost good enough for using Signal in the terminal. Support for stickers is however missing, and signal-cli is a Java daemon which uses up more memory than it should. Not as much as the Electron client, but it could still be better. Ideally, a new client would be built on the C implementation of Signal's protocol</a>. It'd probably have to reimplement a fair amount of the Electron client's features. Resources</h2> Self-Hosted Podcast</a>: Great for getting ideas about the what, why, and how of self-hosting.</li> </ul>

The Defra pet database system is broken

Cat pictures in return for your time</h2>
</p>

A Bandcamp plugin for Grayjay

Linking in C & C++

Running NixOS on the Librem 5

A Journey into Nix and NixOS

Book review: Meditations, Marcus Aurelius Antoninus

Personal thoughts</h2>

Splitting up Liquibase JSON changelogs

Git from first principles

Exploring history and branches</h2>

Do an internship

Diagnosing issues with tcpreplay & tcprewrite

Good design patterns: the command palette

Domain name hacks

Software I use

Personal computers</h2>
You can find most of my configuration files here</a> .</p>

Self-hosting</h2>

Raspberry Pi</h2>

Android smartphone</h2>

DAVx⁵</a></h3>
A CalDAV/CardDAV/WebCal client, useful for hooking up your phone's calendar and contacts to Nextcloud</a>.</p>

Nextcloud</a></h3>
I use the official Nextcloud and Nextcloud news apps. The unofficial notes app</a> provides a better experience when editing notes, so I prefer it over the normal Nextcloud app.</p>

Media server with Kodi</h3>
Ideally I'd have a small computer with mass storage that could act as a server within my LAN, and could store media that I want to keep at hand but not necessarily on my PC. Combined with Kodi</a>, such a computer could provide a great home-entertainment system.</p>

Resources</h2>

Self-Hosted Podcast</a>: Great for getting ideas about the what, why, and how of self-hosting.</li> </ul>

Avery Vigolo

A simple C++ module system with CMake

Avery Vigolo

A simple C++ module system with CMake

The Defra pet database system is broken

Cat pictures in return for your time</h2> </p>

A Bandcamp plugin for Grayjay

Linking in C & C++

Running NixOS on the Librem 5

Introduction</h2> So, I got a used Purism Librem 5. Bought it used with the grand idea of actually doing something useful: contributing to the mobile GNU/Linux ecosystem.</p>

Epiphany</h3> The app initially refused to launch, dying immediately. Running it through the terminal, it complained about not being able to write to ~/.local/state</code>. For some reason, (possibly from nixos-install?) this was owned by root. I chown -R</code>ed the directory to resolve that.</p>

A Journey into Nix and NixOS

Building a QEMU image</h1> I find a couple of guides for making VMs with Nix:</p> https://gist.github.com/tarnacious/f9674436fff0efeb4bb6585c79a3b9ff</li>

Quick diversion: setting up a Nix formatter</h2> Initially running nix fmt</code> gives a somewhat helpful (but not exactly user-friendly) error:</p>

Book review: Meditations, Marcus Aurelius Antoninus

Personal thoughts</h2>

Splitting up Liquibase JSON changelogs

Git from first principles

Exploring history and branches</h2>

Referring to commits by ancestry</h3> You can also refer to commits from their descendants in a relative manner, which saves you the hassle of searching for names when dealing with the recent ancestors of a ref:</p>

Do an internship

Diagnosing issues with tcpreplay & tcprewrite

Good design patterns: the command palette

Cat pictures in return for your time</h2>
</p>