Linux Zero Copy


Purpose

The purpose of this document is to highlight some of the aspects of various Linux subsystems (virtual memory, virtual file system) and how they relate to storage subsystems (file systems and block device drivers). Specifically, the goal is to provide enough information to intelligently design inter-process communication and logging software.

Virtual Memory

Much of this section is summarized from the very good introduction on The Linux Documentation Project.

The virtual memory subsystem presents a view of system memory that is larger than the physical memory available. In addition it provides a number of other features:

  • Large address spaces
  • Process isolation and protection
  • Memory mapping of files or devices
  • Allocation / division of physical memory
  • Shared memory

Abstract Model

The address space of a process is the set of all addresses representable in a pointer (i.e. a 32 or 64 bit word). In userspace, all memory accesses are made against virtual addresses. A virtual address within the process address space is translated to physical addresses by lookup tables managed by the kernel. In order to keep these lookup tables efficient, they operate on fixed sized regions of memory, one unit of which is called a page. The lookup tables are called page tables. On x86 and arm the page size is 4kb (4096 bytes).

Virtual addresses are composed of two fields: frame number and offset. The frame number is the table entry used to lookup the physical page and the offset is where within the physical page the address refers to.

posts/2018/linux_zero_copy/abstract_vm_model.svg

Abstract model of Virtual to Physical address mapping

The page table keeps track of some metadata associated with each mapping:

  • Is this page table entry valid
  • The physical page number this entry points to
  • Access control: read, write, executable

The physical processor is able to use the page table that is managed by the kernel. When a program attempts to access memory the processor will lookup the physical page from the table and (if valid) complete the access operation. If the table entry is not valid, it notifies the kernel by issuing a page fault.

Demand Paging

Given that each process has it's own address space the amount of addressable virtual memory in a multiprocess system can be much greater than the physical memory. Linux implements a number of strategies to efficiently utilize this limited physical memory. One of those strategies is demand paging of process images.

When a program (in ELF format) is first started, the ELF interpreter maps the program file into memory at which point we refer to it as the process image. This mapping is initially unresolved (except for the initial portion) in the page table and physical memory is not yet dedicated to the process image. Instead, pages are filled on demand in response to page faults encountered as the program image is accessed.

When the processor encounters a memory access in the program flow (it may need to fetch the next instruction, jump to a different instruction, or fetch or write to memory) it resolves the virtual address to a physical address through the processes page table. As mentioned above, if there is no entry in the process page table for that virtual address or if the entry is invalid the processor issues a page fault and control of the processor moves to the kernel.

For example, in the figure above there is no entry in process 1's page table for virtual page frame number 2 and so if process 1 attempts to read from an address within virtual page frame number 2 the processor cannot translate the address into a physical one.

If the virtual address of the access that induced the page fault is invalid, then the program is trying to read or write to an address that it has not configured. This is known as a segmentation fault. The kernel will signal the program with SIGSEGV and the program counter will jump to the signal handler (usually resulting in process termination).

If the virtual address of the access is valid but there is no physical page backing it, the kernel must assign a physical page to that virtual page, and then fill that page with the program contents read off from disk. In general this is a time consuming process and so this is an opportunity for the scheduler to service some other process on the processor that issued the fault. Once the fetched page is copied to physical memory and an entry is added to the page table, the process is restarted at the faulting instruction. This time the virtual memory address is successfully translated to a physical address by the processor and the program continues.

Shared Virtual Memory

Despite nominally providing process isolation, virtual memory provides the facilities for processes to share memory. The same physical page of memory may appear in multiple page tables. If it does, then that physical page of memory will be shared between those two processes. Note that the virtual address of that page within those two processes is generally not the same.

There are several ways of getting the same physical page mapped into the page table for two different processes:

  • POSIX shared memory.
  • System V shared memory
  • A shared, anonymous memory map inherited through fork()
  • A file-backed memory map on a common file

This is in addition to (e.g. threads):

  • clone() with flags indicating shared address space

Details of these mechanisms are discussed in a later section.

Linux Page Cache

In general, all reads and writes to real files in Linux go through the Linux page cache. This is a fundamental aspect of Linux performance and has far-reaching implications, including some that we may exploit for optimization.

The Linux page cache is an in-memory generally write-back/read-through cache for file data. When data is read from a regular file it is first moved to page cache, and then made available through the filesystem driver. When data is written to a file, it is first copied to the page cache, and then flushed out to storage at some point later.

The purpose of the page cache is to speed up access to files on storage media. Files are generally read in a page at a time and these pages are stored in the page cache.

posts/2018/linux_zero_copy/page_hash_table.svg

The Linux Page Cache

Each file in Linux is identified by an data structure called an inode, and in Linux pretty much everything is a file (and so has an inode associated with it). When a page from a memory mapped file is read, it is processed through the page cache. If the cache is hot the page is served out of the cache. Otherwise a physical page is allocated and the filesystem or storage driver is informed to fill the page.

Pages filled in the page cache generally stay resident until something other demand pushes them out. This is of particular note because, in general, most of the physical memory is in use on Linux (by at least the page cache).

Recovering physical memory

The Linux kernel attempts to keep a pool of physical memory available for future use. The configurable behavior of this pool has two relative parameters:

  • high water mark
  • low water mark

If the amount of physical memory available is greater than the high water mark then the kernel does nothing at all. Anything currently paged into the page cache is left there indefinitely.

Between the high water mark and low water mark the kernel begins to take action. It will start to evict pages out of physical memory. Below the low water mark the kernel gets more aggressive. The difference between the two is the number of pages the kernel will try to free during each attempt.

posts/2018/linux_zero_copy/kswapd_watermarks.svg

kswapd decision points. (a): with lots of physical memory available, the swap daemon doesn't do anything. (b): when available memory drops below the high water mark, the swap daemon attempts to free a couple of pages per timer tick. (c): when available memory drops below the low water mark, the daemon attempts to free more pages per tick.

The whole process is done by the a kernel thread called the kernel swap daemon (kswapd). It is serviced on a timer and at each service it looks at the number of free pages and takes action.

When the swap daemon decides to try and free memory it first looks for page cache entries that can be discarded. It does this by walking around the page cache and inspecting some fixed number of pages at each iteration (clock algorithm), looking for any pages that can be discarded. A page is discardable if the page is not mapped into any process address space.

If the swap daemon doesn't recover enough pages through discarding of disk cache it will then attempt to swap out or discard mapped pages. It looks only at processes that are swappable (some are not), and that have swappable pages. A page can be locked removing it from the candidate pool of swappable pages. If disk swap is enabled, the swap daemon will consider swapping it out to swap file only if it cannot be recovered in another way. Demand-paged program storage, for instance, can be discarded without swapping because the data can be read back from disk if it's needed again later. The swap daemon will preferentially page out old pages versus those that were used recently.

Virtual File System

Again, much of this section is summarized from the very good introduction on The Linux Documentation Project

The Linux Virtual File system (VFS) allows the operating system to interact with heterogeneous storage media utilizing a wide array of different filesystems. Filesystem drivers in Linux translate VFS interactions to filesystem specific interactions with the underlying storage media.

posts/2018/linux_zero_copy/virtual_filesystem.svg

Schematic of the Linux virtual filesystem and caching.

The basic building block of the Linux VFS is the inode. Every file in the filesystem is described by one and only one inode. They describe both the contents and topology of the VFS. inodes and directory contents are cached in the page cache like file contents, though these cache entries are not 1-1 mappings with data on the block-device of the storage medium. Rather, they are translated by the filesystem driver when they are read in. Never-the-less, they still are generally discardable cache entries as the data can always be restored by reading back the relevant blocks of the underlying storage medium (through the translation layer of the filesystem driver).

A filesystem is basically two things:

  1. a specification for how file contents, metadata, and directory information are laid out on a continuous storage medium
  2. a driver software which interprets this specification and provides a consistent API for the kernel to interact with.

Storage media are block devices which are represented in Linux as files. Like other files they get an inode representation in the VFS and reads and writes to these files are cached in the page cache. When the filesystem driver reads data from the block device to, for instance, enumerate inodes or directory entries, the entire block is pulled into the page cache and then the exact data needed is read, interpreted, and used to fill inode and directory structures. These structures are themselves stored within pages of memory pulled from the page cache pool and are subject to cache rules.

Normally read() and write() act similarly. When the userspace read()s data from a file, the filesystem driver copies data from the block device into the userspace buffer for the read.

Memory-mapped files are (potentially) dealt with a little differently. If the file contents are stored page-aligned and byte-for-byte on the block device (they are for a sane filesystem) then the filesystem driver can implement an optimization informing the kernel to map the existing cache pages directly into the process page table.

Process Memory

The fundamental API with which a userspace program interacts with the kernel virtual memory subsystem is through the mmap(2) system call (and it's glibc wrapper function):

void *mmap(void *addr, size_t length, int prot, int flags,
           int fd, off_t offset);
int munmap(void *addr, size_t length);

From the Linux manual:

mmap() creates a new mapping in the virtual address space of the calling process. The starting address for the new mapping is specified in addr. The length argument specifies the length of the mapping (which must be greater than 0).

mmap() can be used to map the contents of a file into the process address space. The process can then read from or write to the file by simply dereferencing a pointer. In particular:

The contents of a file mapping (as opposed to an anonymous mapping; see MAP_ANONYMOUS below), are initialized using length bytes starting at offset in the file (or other object) referred to by the file descriptor fd. offset must be a multiple of the page size as returned by sysconf(_SC_PAGE_SIZE).

Of particular note is how program instructions are accessed, which was introduced previously in the discussion on the virtual memory subsystem. Consider the execution of a program in the Executable and Linker File (ELF) format. When a program is executed with the exec(3) family of functions (and the underlying system call) Linux will replace the current process image with the system interpreter by mapping it into process address space and moving the program counter to first address in the interpreter program. The interpreter then maps the ELF file into memory, parses out some of the metadata, and then jumps to the start of the program in the mapped file. We often refer to the ELF file (as mapped into memory) as the program image. When the interpreter maps this file into memory it does so as an executable read-only mapping.

Calling mmap() with flags |= MAP_ANONYMOUS is how a process maps general purpose physical memory into it's address space. Specifically:

The mapping is not backed by any file; its contents are initialized to zero. The fd argument is ignored; however, some implementations require fd to be -1 if MAP_ANONYMOUS (or MAP_ANON) is specified, and portable applications should ensure this. The offset argument should be zero. The use of MAP_ANONYMOUS in conjunction with MAP_SHARED is supported on Linux only since kernel 2.4.

This is the underlying mechanism of how memory allocators (i.e. malloc()) work. They call mmap() to map physical pages into the process address space, then they add additional metadata and various global data structures to provide a higher level interface on top of that. Note that the glibc implementation of malloc() never calls munmap(). Any free()ed memory is kept mapped for reuse in a later malloc() call.

Calling mmap() with a file descriptor of an open file handle will initialize the map with the contents of the file, starting at offset. Calling with flags |= MAP_SHARED means that updates to the mapping are visible to other processes with the same region mapped into their address space, and (in the case of a file mapping), writes the the map are carried through to the underlying file. Specifically:

MAP_SHARED Share this mapping. Updates to the mapping are visible to other processes mapping the same region, and (in the case of file-backed mappings) are carried through to the underlying file. (To precisely control when updates are carried through to the underlying file requires the use of msync(2).)

A shared mapping (whether anonymous or file-backed) allows the physical page to reside in the page table for more than one process.

Process shared memory without the posix API

As mentioned above, there are several ways by which multiple process may share memory. The most common mechanism is through the posix shared memory API, though there are others, such as the older (less performant) System V API.

One less used alternative is to share memory through a common process launcher. Let's say that we have two programs foo and bar that we wish to share a common region of memory. We create a launcher program launcher that creates an MAP_ANONYMOUS | MAP_SHARED file mapping, and then fork()s twice. One child calls exec("foo") and the other exec("bar"). When these two programs start, they will start up with the shared region pre-mapped.

Process shared memory with a file-backing: Zerocopy

Processes may also share memory by mapping the same region of the same file into their address space using MAP_SHARED. The mapped regions of memory will refer to the same physical pages. Writes to a page by one process are visible through reads by any other processes.

More importantly, however, is the fact that the physical pages behind the backing are the same physical pages as the disk cache. This fact can be exploited in certain use cases to optimize data flow throughout the system.

Consider this scenario: We have an embedded Linux device receiving images from a MIPI camera through the Video For Linux (V4l2) API, saving them to a file, and also doing some useful work on them (perhaps analyzing them and recording statistics). If we're not careful, may end up copying the images multiple times.

posts/2018/linux_zero_copy/mipi_scenario_a.svg

Simple embedded device application. One process reads camera images from a device, while another process writes those images to storage.

A simple choice of options for how we may implement this scenario is the following:

  • Split the task into to two processes: one to manage the camera and read data in. A second to do useful work, and write data out.
  • process 1 allocates image buffers using malloc() (or new() in c++) and registers them with v4l2 (as V4l2_USER_POINTER structures).
  • When images are received at process 1, send() them over a Unix domain socket to process 2
  • recv() the image at process 2, do the work on the image, and then write() the data to an open file descriptor.

This choice may result in up to 5 separate instances of the same data in memory! A more informed choice may reduce this to only a single instance. Because we never copy this data, this is referred to as a zerocopy optimization.

First, lets illustrate where all these extra instances are coming from.

posts/2018/linux_zero_copy/mipi_scenario_b.svg

Lower-level schematic representation of data flow for the camera to storage application example.

First, the v4l2 driver may not be able to fill the userspace buffer directly due to page alignment or size restrictions. Instead, it may be forced to map and fill a buffer of it's own, and then copy that into the buffer that program 1 allocated. Second, when writing to a socket, data is copied to a network-stack buffer in the kernel until it can be delivered. When process 2 recv()s from the socket, the data is copied again into a process 2 allocated buffer. Finally when process 2 calls write() the data is copied from the userspace buffer into page cache and asynchronously flushed out to the storage medium through the storage driver.

An zero-copy implementation of the same application might be as follows:

  • Process 1 and 2 both mmap() the same region of the output file into their address space.
  • Process 1 registers the buffer with v4l2 (as V4l2_MMAP structures).
  • When an image is received at process 1 it notifies process 2 by sending a very small notification message over a Unix domain socket.
  • Process 2 dereferences it's pointer to the mapped data and directly accesses the image memory.
posts/2018/linux_zero_copy/mipi_scenario_c.svg

Optimized scenario.

In this implementation only one instance ever resides in physical memory. The same physical memory pages are mapped through page tables (or unmapped to physical addresses, in the case of kernel context) to four different pieces of software:

  • v4l2 driver thread in the kernel
  • process 1
  • process 2
  • storage driver

The v4l2 driver fills the physics memory with image data, and then notifies process 1. Process 1 then notifies process 2. Process 2 may access the data by simply addressing it from within it's own address space. The storage driver will asynchronously and opportunistically flush dirty file pages to storage in the background.

A note on O_DIRECT

In the above zerocopy discussion we eliminate additional copies by taking advantage of the existing page cache copy of the data. Another option that can reduce data copies during write is to write to a file opened with O_DIRECT. The exact semantics of O_DIRECT depends on the implementation of the filesystem driver, but in general:

O_DIRECT (since Linux 2.4.10) Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user- space buffers. The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred. To guarantee synchronous I/O, O_SYNC must be used in addition to O_DIRECT.

Usually what this means is that a write() does not copy data to page cache. Instead, the file system driver will immediately attempt to flush data to the underlying storage medium. As mentioned in the manual quote above, in general this will degrade performance. Specifically, the filesystem and storage drivers may be forced to service the write() immediately, rather than interleaving the data flush with other system operations in an optimal way.

Filesystem Reliability

While the Linux page cache is a significant boon to performance in general, one penalty of this system is the reliability of storage on an unclean shutdown or sudden loss of power situation. Any number filesystem changes may appear to a userspace program to have been written to disk, when in fact they have only been written to disk cache. This includes:

  • new files
  • deleted file
  • new data written to a file
  • file changed size

Consider specifically the case of the ext2 filesystem, for which the inode structure is illustrated below.

posts/2018/linux_zero_copy/ext2_inode.svg

EXT2 filesystem inode structure and how it points to file data.

The inode structure forms the root of an unbalanced tree of data blocks containing the actual file data. For small files all the data blocks are referenced directly in the inode data structure. For larger file, the first segment is referenced directly by the inode, and then for later segments the inode points to another data structure, which then points to the data blocks.

Now consider a process which performs following steps (indicated with their libc function calls):

  1. open() a new file (i.e. with flags |= O_CREAT)
  2. periodically write() a chunk of data to the file
  3. close the file

Let's walk through the sequence of filesystem operations that will occur underlying the hood.

  1. When the file is created, a new inode structure is allocated for the file, then a directory entry is added to the parent directory file pointing to this new inode.
  2. Each time we write to the file, the filesystem driver will first check to see if there is an unfilled block already allocated to hold the data. If not it will find a free block and add a pointer to it in the inode and update it's accounting of the file size. Then it will fill that block with data and update the file modification time.
  3. As the file gets larger, we'll overflow the direct blocks and the filesystem driver will need to allocate an indirect block structure, add a pointer to the inode structure, and then allocate a block and add a pointer to the indirection structure.

Each of these modifications is essentially a write to some block of the underlying block device. Each of these writes are subject to page caching and so the actual write really just happens in memory. At some point in the future those dirty pages are flushed out to disk and persisted. In an unclean shutdown or sudden power loss any write waiting in page cache to be flushed will not be persisted to disk. This can include:

  • file data
  • inode data
  • indirect block
  • entry data
  • directory content data.

Modern journaled filesystems (like ext4) deal with this situation by recording in a "journal" all filesystem modifications in-sequence. It ensures these journal entries are flushed to disk before the corresponding data changes are, so that any outstanding changes can be completed by the filesystem driver the next time it is started.

However, it is important to recognize the nature of these interactions and design filesystem access patterns to best account for what will happen on a sudden power loss. In the example above each incremental write may lead to a significant change of filesystem metadata. A sudden power loss may lose those metadata changes, those data changes, or both. A better design for such a process would be to open the file for writing, and then pre-truncate it to the desired size (if possible) or to some reasonably conservative estimate of it's size. This way the filesystem updates all it's metadata and storage allocations up front and it doesn't have to do any of those updates on the fly. A sudden power loss might still lose data of course, but it will be limited to only the unflushed pages in the page cache and not due to missing filesystem metadata or pointers.

A Note on Memory Usage Metrics

When it comes to system-level performance monitoring there are a couple of top level metrics that are often tracked:

  • CPU utilization
  • GPU (or other co-processor) utilization
  • memory utilization

Given the discussion of the page cache and virtual memory systems above, we can now discuss how to measure and meaningfully understand memory utilization on a Linux system. Most userspace tools (like free, for instance) get their information from /proc/meminfo, a virtual file served up from the kernel containing information about memory usage. Due to the Linux page cache and demand-paged memory access, nearly all physical memory (up to the kswapd low-watermark) on a Linux system will be in use. When we talk about "free" memory on a Linux system we are usually referring to the MemAvailable entry of /proc/meminfo.

MemAvailable %lu (since Linux 3.14) An estimate of how much memory is available for start‐ ing new applications, without swapping.

Pages that are reclaimable without swapping include unlocked pages of both the disk cache and buffer cache, including demand-paged file data which is currently resident. Linux's accounting for this value, however, is inexact. It may not know whether a page memory can really be reclaimed until the swap daemon gets around to actually trying to free it. In addition, the fact that this amount of memory can be reclaimed doesn't mean that it can be reclaimed and repurposed instantaneously, and some actions may fail due to low memory if the swap daemon can't find enough pages to release fast enough.

Nvidia Kernel Channel


Background

Relatively rarely but often enough to be a problem we see a kernel NULL pointer dereference or failed paging request. The error occurs in one of three locations (see failure points below). The problem seems to only occur on some exceptional condition (i.e. syncpt timeout). If the stream is completely nominal then we do not see this issue.

Hypothesis

The nvidia v4l2 camera driver contains a race condition in tegra/camera/channel.c. The struct tegra_channel contains a four-element ring buffer of struct vb2_buffer objects. The driver utilizes two kernel threads: the capture thread starts a frame capture, and the release thread concludes a frame capture. Nominally they operate on a shared ring buffer. One thread writes elements into the buffer, and the other thread reads elements out of the buffer. However both threads have some error condition where they reset the buffer, and I believe that this unsynchronized reset is the problem. My hypothesis is that one thread resets the buffer while another thread is in the process of reading/writing to it.

In particular it appears that the purpose of this buffer is to indicate "this is the set of frames to discard if we need to reset". It includes the set of frames that are in release as well as potentially one frame that is on it's way to begin capture.

Solution

The requirement satisfied by the buffer solution is the following: If an error is returned at either the soc_camera_capture_frame() or soc_camera_release_frame() calls, then the VI will be reset and so it doesn't make sense to keep processing frames. Some specific examples:

  • If there is a syncpt timeout error in the release thread while waiting for the frame to finish then we want the capture thread to reset, and any frames currently in flight we want to be released immediately without passing them to the VI.
  • If there is a frame start error in the capture thread, then will reset the VI so we want to flush any frames that are "in flight" without waiting for the VI to finish filling them.

One way of solving this is to keep track of an atomic reset_version integer. When the capture thread restarts the stream, it increments this count. Each time it starts capture on a buffer it stamps the buffer with the current reset_version. Ever time the release thread starts to wait on a frame, it verifies that the reset_version of the buffer matches the current reset_version. If it is different, it flushes the buffer without waiting for it to finish receiving (because it was queued before the VI was reset). Every time the capture thread preps a buffer for capture it checks to see if the reset_version has changed. If it has, then it will reset the VI first.

Failure points

There are two kernel threads. One to capture data to buffer, and a second to release the buffer. Presumably they do their work at start of frame and at end of frame.

The entry points for the two threads are

  • tegra_channel_kthread_capture_start():
  • tegra_channel_kthread_release():

However note that some of the functions executed here might actually be executed in a third thread. For instance (stack 1) to indicate it's actually the userspace thread calling into an ioctl that is leading to the call.

The crash seems to generally occur in one of these three places. My hypothesis is that there is race condition, due to the fact that we see the error consistently on a failed stream closure.

stack 1:

[ 1400.657205] [<ffffffc000750044>] tegra_channel_queued_buf_done+0x64/0x160
[ 1400.665275] [<ffffffc000750354>] tegra_channel_stop_streaming+0x54/0xc8
[ 1400.673150] [<ffffffc00073d5b0>] __vb2_queue_cancel+0x30/0xa8
[ 1400.680135] [<ffffffc00073d834>] vb2_streamoff+0xac/0x124
[ 1400.686744] [<ffffffc00073f014>] vb2_ioctl_streamoff+0x44/0x50
[ 1400.693768] [<ffffffc000729ba0>] v4l_streamoff+0x1c/0x24
[ 1400.700246] [<ffffffc00072dd90>] __video_do_ioctl+0x174/0x284
[ 1400.707136] [<ffffffc00072dafc>] video_usercopy+0x1e4/0x2e8
[ 1400.713831] [<ffffffc00072dc14>] video_ioctl2+0x14/0x1c
[ 1400.720155] [<ffffffc000728990>] v4l2_ioctl+0x70/0x134
[ 1400.726365] [<ffffffc0001be4c4>] vfs_ioctl+0x1c/0x44
[ 1400.732373] [<ffffffc0001bf194>] do_vfs_ioctl+0x220/0x22c
[ 1400.738795] [<ffffffc0001bf274>] SyS_ioctl+0xd4/0x17c

stack 2:

May  5 10:01:37 spitmule3 kernel: [ 7675.570947] [<ffffffc00074fb7c>] dequeue_buffer+0x3c/0x7c
May  5 10:01:37 spitmule3 kernel: [ 7675.576347] [<ffffffc000750ed8>] tegra_channel_kthread_capture_start+0x154/0x1c4
May  5 10:01:37 spitmule3 kernel: [ 7675.583741] [<ffffffc0000d04b8>] kthread+0xc0/0xc8

stack 3:

May  4 09:42:21 deadfire3 kernel: [ 6656.704312] [<ffffffc00074fd94>] free_ring_buffers+0x3c/0xd0
May  4 09:42:21 deadfire3 kernel: [ 6656.709971] [<ffffffc00074ffe8>] tegra_channel_release_frame+0xa0/0xac
May  4 09:42:21 deadfire3 kernel: [ 6656.716495] [<ffffffc00075055c>] tegra_channel_kthread_release+0x178/0x1a4
May  4 09:42:21 deadfire3 kernel: [ 6656.723366] [<ffffffc0000d04b8>] kthread+0xc0/0xc8

Program flow

Life of a buffer

posts/2018/nvidia_kernel_channel/nvidia_kernel_channel.svg

Life of a buffer

  • tegra_channel_buffer_queue(): inserts a buffer into capture
    • buf_queue member of vb2_ops, I believe this is called throught he v4l2 ioctl to enqueue a buffer.
  • dequeue_buffer(): removes a buffer from capture and adds it to buffer
  • tegra_channel_ring_buffer(): nominally adds a buffer release, but may also call free_ring_buffers()
  • tegra_channel_kthread_release: removes a frame from release
  • free_ring_buffers(): removes a buffer from buffer and marks it as done, I believe returning it to v4l2 and allowing it to be retreived with the v4l2 ioctl to dequeue a buffer.

Capture thread

The capture thread loop is basically:

  • dequeue_buf(): Retrieve a buffer that is ready to be filled
  • tegra_channel_capture_frame(chan, buf): Start filling the buffer with frame data.

Release thread

The release thread is basically

  • list_del_init(&buf->queue): remove the buffer from the chan->release list
  • tegra_channel_release_frame(chan, buf): release the frame to userspace

Frame Buffers

The vb2 buffer objects are stored in the buffers field of the struct tegra_channel. This is a fixed size array of four elements. It is used as a ring buffer through the indices save_index and free_index:

  • ``save_index``: The "write" head of the ring buffer. If we need to add an element to the ring this is where we add it, and the increment (with wrap-around).
  • ``free_index``: THe "read" head of the ring buffer. If we want to remove an element of the ring this where we remove from, and increment (with wrap-around).

Possible races

``free_ring_buffers()``, called by:

  • capture thread:
    • stack 1 (CAPTURE_ERROR)
      • free_ring_buffers()
      • tegra_channel_ring_buffer()
      • tegra_channel_capture_frame()
      • tegra_channel_kthread_capture_start()
    • stack 2 (capture state unspecified)
      • free_ring_buffers()
      • tegra_channel_ring_buffer()
      • tegra_channel_capture_frame()
      • tegra_channel_kthread_capture_start()
  • release thread
    • stack 1 (CAPTURE_SUCCESS):
      • free_ring_buffers()
      • tegra_channel_release_frame()
    • stack 2 (CAPTURE_TIMEOUT):
      • free_ring_buffers()
      • tegra_channel_ring_buffer()
      • tegra_channel_release_frame()
  • v4l userspace thread:
    • stack 1 (CAPTURE_GOOD)
      • free_ring_buffers()
      • tegra_channel_ring_buffer()
      • tegra_channel_capture_done()
      • tegra_channel_stop_streaming()
    • stack 2 (always)
      • free_ring_buffers()
      • tegra_channel_stop_streaming()

``add_buffer_to_ring()``, called by:

  • stack 1
    • add_buffer_to_ring()
    • dequeue_buffer()
    • tegra_channel_capture_done()
    • tegra_channel_stop_streaming()
  • stack 2
    • add_buffer_to_ring()
    • dequeue_buffer()
    • tegra_channel_kthread_capture_start()

Some relevant notes:

  • tegra_channel_stop_streaming() calls tegra_channel_stop_kthreads() before it calls free_ring_buffers() and tegra_channel_stop_threads() does wait on a completion indicating the thread has exited before it continues.
  • that tegra_channel_ring_buffer is called by both tegra_channel_capture_frame() and tegra_channel_release_frame() so there is a second path to free_ring_buffers() from the release thread.
  • tegra_channel_ring_buffer() takes a int state argument, which is not used. I suspect the intent was to communicate the capture state, but this is stored instead in chan->capture_state (where chan is a struct tegra_channel).
  • Unable to handle kernel paging request at virual address... is likely an indicator of a corrupt stack. Perhaps a buffer overflow. The dead address prefix likely indicates that the pointer was cleared out or invalidated somehow (i.e. free()ed?
  • Several functions will set chan->capture_state to something other than CAPTURE_GOOD but there are no functions in channel.c that set it to CAPTURE_GOOD. This is done instead in vi2_channel_capture_frame() defined in vi2_fops.c. This function is assigned as the soc_channel_capture_frame member of the chan->fops vtable which is called at the end of tegra_channel_enable_stream().
  • If free_ring_buffers() is called from tegra_channel_ring_buffer() or tegra_channel_stop_streaming() then the number of buffers to free is chan->num_buffers. If called from tegra_channel_release_frame() then the number of buffers to free is one.

Some notes on kernel code

Freezing of tasks

From here

This is how user space processes and some kernel threads are controlled during hybernation or system-wide susped.

All freezable tasks must react to that by calling try_to_freeze(), which results in a call to __refrigerator() (defined in kernel/freezer.c), which sets the task's PF_FROZEN flag, changes its state to TASK_UNINTERRUPTIBLE and makes it loop until PF_FROZEN is cleared for it.

Sleeping

From here

The preferred alternative is wait_event_interruptible, which can be interrupted by signals. This version returns an integer value that you should check; a nonzero value means your sleep was interrupted by some sort of signal, and your driver should probably return -ERESTARTSYS.

In particular tegra_channel_kthread_capture_start sleeps until the following condition is true:

!list_empty(&chan->capture) || kthread_should_stop()

Which I interpret to mean, in english "Sleep until there is a buffer available to fill we are supposed to die."

Completion

From here

Completions are a code synchronization mechanism which is preferable to any misuse of locks. Any time you think of using yield() or some quirky msleep(1) loop to allow something else to proceed, you probably want to look into using one of the wait_for_completion*() calls instead. The advantage of using completions is clear intent of the code, but also more efficient code as both threads can continue until the result is actually needed.

Spin Locks

From here

Spinlocks are simple in concept. A spinlock is a mutual exclusion device that can have only two values: "locked" and "unlocked." It is usually implemented as a single bit in an integer value. Code wishing to take out a particular lock tests the relevant bit. If the lock is available, the "locked" bit is set and the code continues into the critical section. If, instead, the lock has been taken by somebody else, the code goes into a tight loop where it repeatedly checks the lock until it becomes available. This loop is the "spin" part of a spinlock.

List Operations

list_for_each_entry_safe doesn't really indicate whether or not the list may be empty. I think the answer is no (otherwise, it's not a well formed list). In particular, note that tegra_channel_kthread_release() and dequeue_buffer() both check !list_empty(...) instead of just iterating under the assumption that it is non-empty. list_entry get the struct for this entry

Channel State

The following is defined in mc_common.h. Comments have been moved from the header to the field for readability.

struct tegra_channel
{
  // list entry in a composite device dmas list
  struct list_head list;

  // V4L2 video device associated with the video channel
  struct video_device video;

  // media pad for the video device entity
  struct media_pad pad;

  // pipeline belonging to the channel
  struct media_pipeline pipe;
  struct mutex video_lock;

  // composite device DT node port number for the channel
  struct tegra_mc_vi *vi;
  struct v4l2_subdev *subdev[MAX_SUBDEVICES];
  struct v4l2_subdev *subdev_on_csi;

  struct v4l2_ctrl_handler ctrl_handler;
  struct v4l2_pix_format format;
  const struct tegra_video_format *fmtinfo;
  struct mutex stop_kthread_lock;

  unsigned char port[TEGRA_CSI_BLOCKS];
  unsigned int syncpt[2][TEGRA_CSI_BLOCKS];
  unsigned int syncpoint_fifo[2][TEGRA_CSI_BLOCKS];
  unsigned int buffer_offset[TEGRA_CSI_BLOCKS];
  unsigned int buffer_state[QUEUED_BUFFERS];
  struct vb2_buffer *buffers[QUEUED_BUFFERS];
  unsigned int timeout;

  // Next buffer slot available for insertion of a new
  // buffer. Incremented by add_buffer_to_ring()
  unsigned int save_index;

  // Next buffer slot containing a buffer ready go be
  // removed. Incremented by free_ring_buffers()
  unsigned int free_index;
  unsigned int num_buffers;
  unsigned int released_bufs;

  // kernel thread task structure for the capture thread
  struct task_struct *kthread_capture_start;

  // kernel thread task structure for the release thread
  struct task_struct *kthread_release;

  // wait queue for capture thread
  wait_queue_head_t start_wait;

  // wait queue for release thread
  wait_queue_head_t release_wait;

  // vb2 buffer queue
  struct vb2_queue queue;

  // allocation context for the vb2 queue
  void *alloc_ctx;

  // list of queued buffers for capture
  struct list_head capture;

  // list of queued buffers for release
  struct list_head release;

  spinlock_t start_lock;
  spinlock_t release_lock;
  struct completion capture_comp;
  struct completion release_comp;

  // csi register bases
  void __iomem *csibase[TEGRA_CSI_BLOCKS];

  // channel buffer stride alignment, default is 64
  unsigned int stride_align;

  // image width alignment, default is 4
  unsigned int width_align;
  unsigned int valid_ports;
  unsigned int total_ports;
  unsigned int numlanes;

  // Tegra IO rail ID of this video channel
  unsigned int io_id;
  unsigned int num_subdevs;

  // v4l2 buffers sequence number
  unsigned int sequence;
  unsigned int saved_ctx_bypass;
  unsigned int saved_ctx_pgmode;
  unsigned int gang_mode;
  unsigned int gang_mode_default;
  unsigned int gang_width;
  unsigned int gang_height;
  unsigned int gang_bytesperline;
  unsigned int gang_sizeimage;

  // formats supported
  DECLARE_BITMAP(fmts_bitmap, MAX_FORMAT_NUM);
  atomic_t power_on_refcnt;
  struct v4l2_fh *fh;

  // bypass flag for VI bypass mode
  bool bypass;
  bool bfirst_fstart;
  enum channel_capture_state capture_state;
  atomic_t is_streaming;
  int requested_kbyteps;
  unsigned long requested_hz;
  int grp_id;

  struct tegra_vi_channel_fops *fops;
};

Also note that the constant QUEUED_BUFFERS is = 4.

Error traces

May  2 14:11:20 deadfire3 kernel: [ 8675.391692] video4linux video3: frame start syncpt timeout!0
May  2 14:11:20 deadfire3 kernel: [ 8675.399203] Unable to handle kernel NULL pointer dereference at virtual address 0000004c
May  2 14:11:20 deadfire3 kernel: [ 8675.411531] pgd = ffffffc00007d000
May  2 14:11:21 deadfire3 kernel: [ 8675.414938] [0000004c] *pgd=000000017fc05003, *pmd=000000017fc06003, *pte=00e0000050041407
May  2 14:11:21 deadfire3 kernel: [ 8675.423853] Internal error: Oops: 96000046 [#1] PREEMPT SMP
May  2 14:11:21 deadfire3 kernel: [ 8675.429421] Enter nvdumper_crash_setup_regs
May  2 14:11:21 deadfire3 kernel: [ 8675.433608] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.433610] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.433611] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.446934] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.451375] Modules linked in: bcmdhd cfg80211 bluedroid_pm
May  2 14:11:21 deadfire3 kernel: [ 8675.457020] CPU: 2 PID: 20545 Comm: vi-output-4, tc Not tainted 3.10.96-skydio-gc145d3a6e8d-dirty #147
May  2 14:11:21 deadfire3 kernel: [ 8675.466318] task: ffffffc0fe172fc0 ti: ffffffc0d6310000 task.ti: ffffffc0d6310000
May  2 14:11:21 deadfire3 kernel: [ 8675.473802] PC is at free_ring_buffers+0x3c/0xd0
May  2 14:11:21 deadfire3 kernel: [ 8675.478420] LR is at tegra_channel_release_frame+0xa0/0xac
May  2 14:11:21 deadfire3 kernel: [ 8675.483902] pc : [<ffffffc00074fe10>] lr : [<ffffffc000750064>] pstate: 60000145
May  2 14:11:21 deadfire3 kernel: [ 8675.491290] sp : ffffffc0d6313d40
May  2 14:11:21 deadfire3 kernel: [ 8675.494609] x29: ffffffc0d6313d40 x28: ffffffc0d6310000
May  2 14:11:21 deadfire3 kernel: [ 8675.499947] x27: 0000000000000001 x26: ffffffc0fc82e1d0
May  2 14:11:21 deadfire3 kernel: [ 8675.505287] x25: ffffffc0d6313e20 x24: ffffffc0000d0ed8
May  2 14:11:21 deadfire3 kernel: [ 8675.510630] x23: ffffffc001003c90 x22: 0000000000000005
May  2 14:11:21 deadfire3 kernel: [ 8675.515969] x21: 0000000000000001 x20: 0000000000000001
May  2 14:11:21 deadfire3 kernel: [ 8675.521310] x19: ffffffc0fc82dbc0 x18: 000000000000013f
May  2 14:11:21 deadfire3 kernel: [ 8675.526649] x17: 0000007f993c130c x16: 0000000000000000
May  2 14:11:21 deadfire3 kernel: [ 8675.531985] x15: 0000000000000000 x14: 0000000000000001
May  2 14:11:21 deadfire3 kernel: [ 8675.537324] x13: 000000000000002b x12: 0000000000000150
May  2 14:11:21 deadfire3 kernel: [ 8675.542664] x11: 0000000000129fbe x10: 00000000ffa722d2
May  2 14:11:21 deadfire3 kernel: [ 8675.548001] x9 : ffffffc0d6313bc0 x8 : 00ffffffffffffff
May  2 14:11:21 deadfire3 kernel: [ 8675.553341] x7 : 0000000034155555 x6 : 00000000000021e2
May  2 14:11:21 deadfire3 kernel: [ 8675.558679] x5 : ffffffffc465ffff x4 : 000000000000c9ff
May  2 14:11:21 deadfire3 kernel: [ 8675.564015] x3 : 00000000000021e3 x2 : 0000000000099090
May  2 14:11:21 deadfire3 kernel: [ 8675.569352] x1 : 000000000009908f x0 : 0000000000000014
May  2 14:11:21 deadfire3 kernel: [ 8675.574691]
May  2 14:11:21 deadfire3 kernel: [ 8675.574691] PC: 0xffffffc00074fd90:
May  2 14:11:21 deadfire3 kernel: [ 8675.579652] fd90  f9400400 b94002a1 97f13b13 f9422a60 f9400400 b9400ea1 97f13b0f 91000694
May  2 14:11:21 deadfire3 kernel: [ 8675.587966] fdb0  910012b5 4b170280 6b0002df 54fffd8c a94153f3 a9425bf5 f9401bf7 a8c47bfd
May  2 14:11:21 deadfire3 kernel: [ 8675.596281] fdd0  d65f03c0 a9bd7bfd 910003fd a90153f3 a9025bf5 aa0003f3 2a0103f4 34000541
May  2 14:11:21 deadfire3 kernel: [ 8675.604594] fdf0  52800035 528000b6 b945da60 9102d800 f8607a60 b948d661 11000422 b908d662
May  2 14:11:21 deadfire3 kernel: [ 8675.612916] fe10  b9003801 b9001015 b9450e62 b9426001 34000041 b9005802 b9491661 7100043f
May  2 14:11:21 deadfire3 kernel: [ 8675.621228] fe30  54000081 b945e261 7100043f 54000088 b945da61 9105a021 b8217a76 b945da61
May  2 14:11:21 deadfire3 kernel: [ 8675.629545] fe50  11000422 b905da62 8b214a61 b945a021 97ffb6de b945da60 71000c1f 54000049
May  2 14:11:21 deadfire3 kernel: [ 8675.637852] fe70  b905da7f b945de60 51000400 b905de60 b945e260 11000400 b905e260 71000694
May  2 14:11:21 deadfire3 kernel: [ 8675.646160]
May  2 14:11:21 deadfire3 kernel: [ 8675.646160] LR: 0xffffffc00074ffe4:
May  2 14:11:21 deadfire3 kernel: [ 8675.651123] ffe4  d63f0060 34000160 aa1303e0 97ffff48 52800040 b9091660 aa1303e0 aa1403e1
May  2 14:11:21 deadfire3 kernel: [ 8675.659443] 0004  910083a2 528000a3 97ffffa6 14000015 f0006400 794bf400 36100160 f0006400
May  2 14:11:21 deadfire3 kernel: [ 8675.667753] 0024  91162000 91014000 91026261 f0004b62 91256042 b0002703 9104e063 9101e063
May  2 14:11:21 deadfire3 kernel: [ 8675.676071] 0044  97efd5f4 b945da60 9105a000 52800081 b8207a61 aa1303e0 52800021 97ffff5d
May  2 14:11:21 deadfire3 kernel: [ 8675.684381] 0064  a94153f3 a8c37bfd d65f03c0 a9ba7bfd 910003fd a90153f3 a9025bf5 a90363f7
May  2 14:11:21 deadfire3 kernel: [ 8675.692691] 0084  a9046bf9 f9002bfb aa0003fa 2a0103f5 9120e017 91212016 9121601b aa1b03e0
May  2 14:11:21 deadfire3 kernel: [ 8675.700999] 00a4  9410778f f9441f40 d10d2013 f9400014 d10d2294 eb0002ff 54000340 d2802019
May  2 14:11:21 deadfire3 kernel: [ 8675.709312] 00c4  d2804018 aa1303e0 2a1503e1 97ffb642 f941aa60 f941a662 f9000440 f9000002
May  2 14:11:21 deadfire3 kernel: [ 8675.717628]
May  2 14:11:21 deadfire3 kernel: [ 8675.717628] SP: 0xffffffc0d6313cc0:
May  2 14:11:21 deadfire3 kernel: [ 8675.722588] 3cc0  00000001 00000000 00000001 00000000 00000005 00000000 01003c90 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.730904] 3ce0  000d0ed8 ffffffc0 d6313e20 ffffffc0 fc82e1d0 ffffffc0 00000001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.739213] 3d00  d6310000 ffffffc0 d6313d40 ffffffc0 00750064 ffffffc0 d6313d40 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.747528] 3d20  0074fe10 ffffffc0 60000145 00000000 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.755847] 3d40  d6313d70 ffffffc0 00750064 ffffffc0 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.764158] 3d60  fc82e41c ffffffc0 014179f0 ffffffc0 d6313da0 ffffffc0 007505d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.772466] 3d80  fc82dbc0 ffffffc0 faff0400 ffffffc0 000021e3 00000000 171ca58d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.780775] 3da0  d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.789088]
May  2 14:11:21 deadfire3 kernel: [ 8675.789088] X5: 0xffffffffc465ff7f:
May  2 14:11:21 deadfire3 kernel: [ 8675.794048] ff7c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.802375] ff9c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.810687] ffbc  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.819006] ffdc  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.827325] fffc  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.835638] 001c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.843951] 003c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.852279] 005c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.860607] 007c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.868929]
May  2 14:11:21 deadfire3 kernel: [ 8675.868929] X9: 0xffffffc0d6313b40:
May  2 14:11:21 deadfire3 kernel: [ 8675.873893] 3b40  d6313ba0 ffffffc0 000813a8 ffffffc0 00000000 00000000 00000400 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.882220] 3b60  d6313b70 ffffffc0 00081214 ffffffc0 d6313d40 ffffffc0 00084c30 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.890536] 3b80  fc82dbc0 ffffffc0 00000001 00000000 d6313d40 ffffffc0 0074fe10 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.898846] 3ba0  40000000 00000000 d6310000 ffffffc0 d6313d10 ffffffc0 00084e1c ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.907164] 3bc0  d6313c00 ffffffc0 0039fe4c ffffffc0 00000002 00000000 fdda90a0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.915484] 3be0  00000000 00000000 0003de80 00000000 d6313d90 ffffffc0 00000005 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.923795] 3c00  d6313ce0 ffffffc0 003a03fc ffffffc0 00000002 00000000 0003de80 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.932117] 3c20  00000014 00000000 0009908f 00000000 00099090 00000000 000021e3 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.940405]
May  2 14:11:21 deadfire3 kernel: [ 8675.940405] X19: 0xffffffc0fc82db40:
May  2 14:11:21 deadfire3 kernel: [ 8675.945450] db40  00000040 00000001 00000001 00000001 00000002 00000000 00000001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.953766] db60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.962082] db80  00008119 00000000 00000000 00000000 fd15d240 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.970388] dba0  00000000 00000000 00000000 00000000 00000004 00000000 012a8368 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.978696] dbc0  00000000 00000000 00000000 00000000 fd59c958 ffffffc0 fc82d298 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.987008] dbe0  fc8280e8 ffffffc0 00000004 00000000 fc82df20 ffffffc0 00010001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.995321] dc00  00000000 00000000 00000000 00010001 00010001 00000000 fc82dfd0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.003639] dc20  fd6ddac0 ffffffc0 00000000 00000000 00000001 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.011955]
May  2 14:11:21 deadfire3 kernel: [ 8676.011955] X23: 0xffffffc001003c10:
May  2 14:11:21 deadfire3 kernel: [ 8676.017003] 3c10  67697326 61773e2d 635f7469 65646c68 00746978 00000000 67697326 72673e2d
May  2 14:11:21 deadfire3 kernel: [ 8676.025326] 3c30  5f70756f 65737772 0000006d 00000000 67697326 72633e2d 675f6465 64726175
May  2 14:11:21 deadfire3 kernel: [ 8676.033642] 3c50  74756d5f 00007865 6d6f682f 6b732f65 6f696479 69766e2f 5f616964 6e72656b
May  2 14:11:21 deadfire3 kernel: [ 8676.041957] 3c70  692f6c65 756c636e 6c2f6564 78756e69 636f692f 65746e6f 682e7478 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.050273] 3c90  6d6f682f 6b732f65 6f696479 69766e2f 5f616964 6e72656b 692f6c65 756c636e
May  2 14:11:21 deadfire3 kernel: [ 8676.058584] 3cb0  6c2f6564 78756e69 6572662f 72657a65 0000682e 00000000 6b736174 7274735f
May  2 14:11:21 deadfire3 kernel: [ 8676.066901] 3cd0  00746375 00000000 7465735f 6469745f 6464615f 73736572 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.075215] 3cf0  2d6d6d26 616d6d3e 65735f70 0000006d 3e2d7826 74696177 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.083537]
May  2 14:11:21 deadfire3 kernel: [ 8676.083537] X24: 0xffffffc0000d0e58:
May  2 14:11:21 deadfire3 kernel: [ 8676.088587] 0e58  910003e1 9272c424 f9400884 f900009f 942a7540 aa0003f5 91006264 f9400e65
May  2 14:11:21 deadfire3 kernel: [ 8676.096899] 0e78  eb05009f 540000e0 f9400483 f90004a3 f9000065 f9000e64 f9000484 14000009
May  2 14:11:21 deadfire3 kernel: [ 8676.105217] 0e98  91002280 f9400681 eb00003f 540000a0 aa1403e0 2a1703e1 aa1603e2 94002435
May  2 14:11:21 deadfire3 kernel: [ 8676.113529] 0eb8  aa1403e0 aa1503e1 942a75a6 a94153f3 a9425bf5 f9401bf7 a8c47bfd d65f03c0
May  2 14:11:21 deadfire3 kernel: [ 8676.121842] 0ed8  a9be7bfd 910003fd f9000bf3 aa0003f3 940045e3 34000100 91006261 f9401262
May  2 14:11:21 deadfire3 kernel: [ 8676.130150] 0ef8  f9400e63 f9000462 f9000043 f9000e61 f9000421 f9400bf3 a8c27bfd d65f03c0
May  2 14:11:21 deadfire3 kernel: [ 8676.138460] 0f18  a9bf7bfd 910003fd aa0003e4 f85f0005 f9400066 52800000 eb0600bf 54000201
May  2 14:11:21 deadfire3 kernel: [ 8676.146766] 0f38  b85f8086 b9400867 6b0700df 54000181 1100fcc0 6b1f00df 1a86b000 13067c00
May  2 14:11:21 deadfire3 kernel: [ 8676.155093]
May  2 14:11:21 deadfire3 kernel: [ 8676.155093] X25: 0xffffffc0d6313da0:
May  2 14:11:21 deadfire3 kernel: [ 8676.160147] 3da0  d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.168486] 3dc0  01003d00 ffffffc0 fc82dbc0 ffffffc0 00750460 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.176804] 3de0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.185115] 3e00  d19a7b50 ffffffc0 00000000 00000000 fe172fc0 ffffffc0 000d0ed8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.193427] 3e20  d6313e20 ffffffc0 d6313e20 ffffffc0 00000000 00000000 00085520 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.201735] 3e40  000d03f8 ffffffc0 d19a7b50 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.210050] 3e60  00000000 00000000 000dcc5c ffffffc0 d6313eb0 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.218364] 3e80  1fe585c0 ffffffc0 fc82dbc0 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.226676]
May  2 14:11:21 deadfire3 kernel: [ 8676.226676] X26: 0xffffffc0fc82e150:
May  2 14:11:21 deadfire3 kernel: [ 8676.231725] e150  00000000 00000000 00000000 00000000 00000005 00000005 00000005 00000005
May  2 14:11:21 deadfire3 kernel: [ 8676.240036] e170  00000004 ffffffc0 faff0400 ffffffc0 d3572800 ffffffc0 fb3e3800 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.248351] e190  00000014 00000000 00000002 fd96d6ac 0269bf88 00000000 fe3fb0c0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.256661] e1b0  fe172fc0 ffffffc0 f82df82d 00000000 fc82e1c0 ffffffc0 fc82e1c0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.264971] e1d0  a1f7a1f7 00000000 fc82e1d8 ffffffc0 fc82e1d8 ffffffc0 00000001 00000017
May  2 14:11:21 deadfire3 kernel: [ 8676.273293] e1f0  00000000 00000000 fc82dfe8 ffffffc0 d51bb540 ffffffc0 00c315d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.281618] e210  00c2fa00 ffffffc0 fc82dbc0 ffffffc0 00000378 00002000 00000000 00000002
May  2 14:11:21 deadfire3 kernel: [ 8676.289932] e230  fab3c800 ffffffc0 faff0400 ffffffc0 d3572800 ffffffc0 fb3e3800 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.298251]
May  2 14:11:21 deadfire3 kernel: [ 8676.298251] X28: 0xffffffc0d630ff80:
May  2 14:11:21 deadfire3 kernel: [ 8676.303298] ff80  cfc4c4cf c2cfc4c2 c3c2cfc4 cbbfc1ce becbbfbe bfbecbbf cbbfbecb becbbfbe
May  2 14:11:21 deadfire3 kernel: [ 8676.311611] ffa0  c1bfccc1 cbbfbfcc becbbfbe c1becac1 c9bebeca bdc8bdbf bcbac7bc c8bfbac7
May  2 14:11:21 deadfire3 kernel: [ 8676.319925] ffc0  bbc8bfbb bfbdc7bf c6bbbdc7 b9c6bbb9 bab8c5ba c6bbb8c5 b8c5bab9 bab9c6bb
May  2 14:11:21 deadfire3 kernel: [ 8676.328234] ffe0  c5bab8c5 b9c6bbb8 b9b7c4b9 c3bab7c4 b8c1b9b9 bab9c3ba c3bab9c3 bac4bbb9
May  2 14:11:21 deadfire3 kernel: [ 8676.336550] 0000  00000002 00000000 ffffffff ffffffff fe172fc0 ffffffc0 000bce50 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.344860] 0020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.353174] 0040  00000000 00000000 00000002 00000002 00000100 00000000 57ac6e9d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.361500] 0060  3f37363d 343e3535 38364038 41393640 36403838 38364038 41393640 36403838
May  2 14:11:21 deadfire3 kernel: [ 8676.369811]
May  2 14:11:21 deadfire3 kernel: [ 8676.369811] X29: 0xffffffc0d6313cc0:
May  2 14:11:21 deadfire3 kernel: [ 8676.374859] 3cc0  00000001 00000000 00000001 00000000 00000005 00000000 01003c90 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.383182] 3ce0  000d0ed8 ffffffc0 d6313e20 ffffffc0 fc82e1d0 ffffffc0 00000001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.391504] 3d00  d6310000 ffffffc0 d6313d40 ffffffc0 00750064 ffffffc0 d6313d40 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.399822] 3d20  0074fe10 ffffffc0 60000145 00000000 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.408149] 3d40  d6313d70 ffffffc0 00750064 ffffffc0 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.416461] 3d60  fc82e41c ffffffc0 014179f0 ffffffc0 d6313da0 ffffffc0 007505d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.424782] 3d80  fc82dbc0 ffffffc0 faff0400 ffffffc0 000021e3 00000000 171ca58d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.433094] 3da0  d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.441410]
May  2 14:11:21 deadfire3 kernel: [ 8676.442904] Process vi-output-4, tc (pid: 20545, stack limit = 0xffffffc0d6310058)
May  2 14:11:21 deadfire3 kernel: [ 8676.450467] Stack: (0xffffffc0d6313d40 to 0xffffffc0d6314000)
May  2 14:11:21 deadfire3 kernel: [ 8676.456212] 3d40: d6313d70 ffffffc0 00750064 ffffffc0 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.464382] 3d60: fc82e41c ffffffc0 014179f0 ffffffc0 d6313da0 ffffffc0 007505d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.472553] 3d80: fc82dbc0 ffffffc0 faff0400 ffffffc0 000021e3 00000000 171ca58d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.480723] 3da0: d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.488893] 3dc0: 01003d00 ffffffc0 fc82dbc0 ffffffc0 00750460 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.497062] 3de0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.505234] 3e00: d19a7b50 ffffffc0 00000000 00000000 fe172fc0 ffffffc0 000d0ed8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.513406] 3e20: d6313e20 ffffffc0 d6313e20 ffffffc0 00000000 00000000 00085520 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.521576] 3e40: 000d03f8 ffffffc0 d19a7b50 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.529748] 3e60: 00000000 00000000 000dcc5c ffffffc0 d6313eb0 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.537918] 3e80: 1fe585c0 ffffffc0 fc82dbc0 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.546088] 3ea0: d6313ea0 ffffffc0 d6313ea0 ffffffc0 00000000 00000000 00000000 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.554258] 3ec0: d6313ec0 ffffffc0 d6313ec0 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.562428] 3ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.570599] 3f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.578769] 3f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.586940] 3f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.595110] 3f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.603282] 3f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.611450] 3fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.619620] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000005 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.627792] 3fe0: 00000000 00000000 00000000 00000000 565c4b45 7f797f5c a79a9499 797fa8a2
May  2 14:11:21 deadfire3 kernel: [ 8676.635960] Call trace:
May  2 14:11:21 deadfire3 kernel: [ 8676.638409] [<ffffffc00074fe10>] free_ring_buffers+0x3c/0xd0
May  2 14:11:21 deadfire3 kernel: [ 8676.644065] [<ffffffc000750064>] tegra_channel_release_frame+0xa0/0xac
May  2 14:11:21 deadfire3 kernel: [ 8676.650587] [<ffffffc0007505d8>] tegra_channel_kthread_release+0x178/0x1a4
May  2 14:11:21 deadfire3 kernel: [ 8676.657455] [<ffffffc0000d04b8>] kthread+0xc0/0xc8
May  2 14:11:21 deadfire3 kernel: [ 8676.662245] Code: f8607a60 b948d661 11000422 b908d662 (b9003801)
May  2 14:11:21 deadfire3 kernel: [ 8676.674126] ---[ end trace 7a9d03f4c84d921b ]---

Userspace thread

May  2 14:11:20 deadfire3 kernel: [ 8675.391692] video4linux video3: frame start syncpt timeout!0
May  2 14:11:20 deadfire3 kernel: [ 8675.399203] Unable to handle kernel NULL pointer dereference at virtual address 0000004c
May  2 14:11:20 deadfire3 kernel: [ 8675.411531] pgd = ffffffc00007d000
May  2 14:11:21 deadfire3 kernel: [ 8675.414938] [0000004c] *pgd=000000017fc05003, *pmd=000000017fc06003, *pte=00e0000050041407
May  2 14:11:21 deadfire3 kernel: [ 8675.423853] Internal error: Oops: 96000046 [#1] PREEMPT SMP
May  2 14:11:21 deadfire3 kernel: [ 8675.429421] Enter nvdumper_crash_setup_regs
May  2 14:11:21 deadfire3 kernel: [ 8675.433608] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.433610] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.433611] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.446934] nvdumper: all registers are saved.
May  2 14:11:21 deadfire3 kernel: [ 8675.451375] Modules linked in: bcmdhd cfg80211 bluedroid_pm
May  2 14:11:21 deadfire3 kernel: [ 8675.457020] CPU: 2 PID: 20545 Comm: vi-output-4, tc Not tainted 3.10.96-skydio-gc145d3a6e8d-dirty #147
May  2 14:11:21 deadfire3 kernel: [ 8675.466318] task: ffffffc0fe172fc0 ti: ffffffc0d6310000 task.ti: ffffffc0d6310000
May  2 14:11:21 deadfire3 kernel: [ 8675.473802] PC is at free_ring_buffers+0x3c/0xd0
May  2 14:11:21 deadfire3 kernel: [ 8675.478420] LR is at tegra_channel_release_frame+0xa0/0xac
May  2 14:11:21 deadfire3 kernel: [ 8675.483902] pc : [<ffffffc00074fe10>] lr : [<ffffffc000750064>] pstate: 60000145
May  2 14:11:21 deadfire3 kernel: [ 8675.491290] sp : ffffffc0d6313d40
May  2 14:11:21 deadfire3 kernel: [ 8675.494609] x29: ffffffc0d6313d40 x28: ffffffc0d6310000
May  2 14:11:21 deadfire3 kernel: [ 8675.499947] x27: 0000000000000001 x26: ffffffc0fc82e1d0
May  2 14:11:21 deadfire3 kernel: [ 8675.505287] x25: ffffffc0d6313e20 x24: ffffffc0000d0ed8
May  2 14:11:21 deadfire3 kernel: [ 8675.510630] x23: ffffffc001003c90 x22: 0000000000000005
May  2 14:11:21 deadfire3 kernel: [ 8675.515969] x21: 0000000000000001 x20: 0000000000000001
May  2 14:11:21 deadfire3 kernel: [ 8675.521310] x19: ffffffc0fc82dbc0 x18: 000000000000013f
May  2 14:11:21 deadfire3 kernel: [ 8675.526649] x17: 0000007f993c130c x16: 0000000000000000
May  2 14:11:21 deadfire3 kernel: [ 8675.531985] x15: 0000000000000000 x14: 0000000000000001
May  2 14:11:21 deadfire3 kernel: [ 8675.537324] x13: 000000000000002b x12: 0000000000000150
May  2 14:11:21 deadfire3 kernel: [ 8675.542664] x11: 0000000000129fbe x10: 00000000ffa722d2
May  2 14:11:21 deadfire3 kernel: [ 8675.548001] x9 : ffffffc0d6313bc0 x8 : 00ffffffffffffff
May  2 14:11:21 deadfire3 kernel: [ 8675.553341] x7 : 0000000034155555 x6 : 00000000000021e2
May  2 14:11:21 deadfire3 kernel: [ 8675.558679] x5 : ffffffffc465ffff x4 : 000000000000c9ff
May  2 14:11:21 deadfire3 kernel: [ 8675.564015] x3 : 00000000000021e3 x2 : 0000000000099090
May  2 14:11:21 deadfire3 kernel: [ 8675.569352] x1 : 000000000009908f x0 : 0000000000000014
May  2 14:11:21 deadfire3 kernel: [ 8675.574691]
May  2 14:11:21 deadfire3 kernel: [ 8675.574691] PC: 0xffffffc00074fd90:
May  2 14:11:21 deadfire3 kernel: [ 8675.579652] fd90  f9400400 b94002a1 97f13b13 f9422a60 f9400400 b9400ea1 97f13b0f 91000694
May  2 14:11:21 deadfire3 kernel: [ 8675.587966] fdb0  910012b5 4b170280 6b0002df 54fffd8c a94153f3 a9425bf5 f9401bf7 a8c47bfd
May  2 14:11:21 deadfire3 kernel: [ 8675.596281] fdd0  d65f03c0 a9bd7bfd 910003fd a90153f3 a9025bf5 aa0003f3 2a0103f4 34000541
May  2 14:11:21 deadfire3 kernel: [ 8675.604594] fdf0  52800035 528000b6 b945da60 9102d800 f8607a60 b948d661 11000422 b908d662
May  2 14:11:21 deadfire3 kernel: [ 8675.612916] fe10  b9003801 b9001015 b9450e62 b9426001 34000041 b9005802 b9491661 7100043f
May  2 14:11:21 deadfire3 kernel: [ 8675.621228] fe30  54000081 b945e261 7100043f 54000088 b945da61 9105a021 b8217a76 b945da61
May  2 14:11:21 deadfire3 kernel: [ 8675.629545] fe50  11000422 b905da62 8b214a61 b945a021 97ffb6de b945da60 71000c1f 54000049
May  2 14:11:21 deadfire3 kernel: [ 8675.637852] fe70  b905da7f b945de60 51000400 b905de60 b945e260 11000400 b905e260 71000694
May  2 14:11:21 deadfire3 kernel: [ 8675.646160]
May  2 14:11:21 deadfire3 kernel: [ 8675.646160] LR: 0xffffffc00074ffe4:
May  2 14:11:21 deadfire3 kernel: [ 8675.651123] ffe4  d63f0060 34000160 aa1303e0 97ffff48 52800040 b9091660 aa1303e0 aa1403e1
May  2 14:11:21 deadfire3 kernel: [ 8675.659443] 0004  910083a2 528000a3 97ffffa6 14000015 f0006400 794bf400 36100160 f0006400
May  2 14:11:21 deadfire3 kernel: [ 8675.667753] 0024  91162000 91014000 91026261 f0004b62 91256042 b0002703 9104e063 9101e063
May  2 14:11:21 deadfire3 kernel: [ 8675.676071] 0044  97efd5f4 b945da60 9105a000 52800081 b8207a61 aa1303e0 52800021 97ffff5d
May  2 14:11:21 deadfire3 kernel: [ 8675.684381] 0064  a94153f3 a8c37bfd d65f03c0 a9ba7bfd 910003fd a90153f3 a9025bf5 a90363f7
May  2 14:11:21 deadfire3 kernel: [ 8675.692691] 0084  a9046bf9 f9002bfb aa0003fa 2a0103f5 9120e017 91212016 9121601b aa1b03e0
May  2 14:11:21 deadfire3 kernel: [ 8675.700999] 00a4  9410778f f9441f40 d10d2013 f9400014 d10d2294 eb0002ff 54000340 d2802019
May  2 14:11:21 deadfire3 kernel: [ 8675.709312] 00c4  d2804018 aa1303e0 2a1503e1 97ffb642 f941aa60 f941a662 f9000440 f9000002
May  2 14:11:21 deadfire3 kernel: [ 8675.717628]
May  2 14:11:21 deadfire3 kernel: [ 8675.717628] SP: 0xffffffc0d6313cc0:
May  2 14:11:21 deadfire3 kernel: [ 8675.722588] 3cc0  00000001 00000000 00000001 00000000 00000005 00000000 01003c90 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.730904] 3ce0  000d0ed8 ffffffc0 d6313e20 ffffffc0 fc82e1d0 ffffffc0 00000001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.739213] 3d00  d6310000 ffffffc0 d6313d40 ffffffc0 00750064 ffffffc0 d6313d40 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.747528] 3d20  0074fe10 ffffffc0 60000145 00000000 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.755847] 3d40  d6313d70 ffffffc0 00750064 ffffffc0 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.764158] 3d60  fc82e41c ffffffc0 014179f0 ffffffc0 d6313da0 ffffffc0 007505d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.772466] 3d80  fc82dbc0 ffffffc0 faff0400 ffffffc0 000021e3 00000000 171ca58d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.780775] 3da0  d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.789088]
May  2 14:11:21 deadfire3 kernel: [ 8675.789088] X5: 0xffffffffc465ff7f:
May  2 14:11:21 deadfire3 kernel: [ 8675.794048] ff7c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.802375] ff9c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.810687] ffbc  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.819006] ffdc  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.827325] fffc  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.835638] 001c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.843951] 003c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.852279] 005c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.860607] 007c  ******** ******** ******** ******** ******** ******** ******** ********
May  2 14:11:21 deadfire3 kernel: [ 8675.868929]
May  2 14:11:21 deadfire3 kernel: [ 8675.868929] X9: 0xffffffc0d6313b40:
May  2 14:11:21 deadfire3 kernel: [ 8675.873893] 3b40  d6313ba0 ffffffc0 000813a8 ffffffc0 00000000 00000000 00000400 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.882220] 3b60  d6313b70 ffffffc0 00081214 ffffffc0 d6313d40 ffffffc0 00084c30 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.890536] 3b80  fc82dbc0 ffffffc0 00000001 00000000 d6313d40 ffffffc0 0074fe10 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.898846] 3ba0  40000000 00000000 d6310000 ffffffc0 d6313d10 ffffffc0 00084e1c ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.907164] 3bc0  d6313c00 ffffffc0 0039fe4c ffffffc0 00000002 00000000 fdda90a0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.915484] 3be0  00000000 00000000 0003de80 00000000 d6313d90 ffffffc0 00000005 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.923795] 3c00  d6313ce0 ffffffc0 003a03fc ffffffc0 00000002 00000000 0003de80 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.932117] 3c20  00000014 00000000 0009908f 00000000 00099090 00000000 000021e3 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.940405]
May  2 14:11:21 deadfire3 kernel: [ 8675.940405] X19: 0xffffffc0fc82db40:
May  2 14:11:21 deadfire3 kernel: [ 8675.945450] db40  00000040 00000001 00000001 00000001 00000002 00000000 00000001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.953766] db60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.962082] db80  00008119 00000000 00000000 00000000 fd15d240 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.970388] dba0  00000000 00000000 00000000 00000000 00000004 00000000 012a8368 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.978696] dbc0  00000000 00000000 00000000 00000000 fd59c958 ffffffc0 fc82d298 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8675.987008] dbe0  fc8280e8 ffffffc0 00000004 00000000 fc82df20 ffffffc0 00010001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8675.995321] dc00  00000000 00000000 00000000 00010001 00010001 00000000 fc82dfd0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.003639] dc20  fd6ddac0 ffffffc0 00000000 00000000 00000001 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.011955]
May  2 14:11:21 deadfire3 kernel: [ 8676.011955] X23: 0xffffffc001003c10:
May  2 14:11:21 deadfire3 kernel: [ 8676.017003] 3c10  67697326 61773e2d 635f7469 65646c68 00746978 00000000 67697326 72673e2d
May  2 14:11:21 deadfire3 kernel: [ 8676.025326] 3c30  5f70756f 65737772 0000006d 00000000 67697326 72633e2d 675f6465 64726175
May  2 14:11:21 deadfire3 kernel: [ 8676.033642] 3c50  74756d5f 00007865 6d6f682f 6b732f65 6f696479 69766e2f 5f616964 6e72656b
May  2 14:11:21 deadfire3 kernel: [ 8676.041957] 3c70  692f6c65 756c636e 6c2f6564 78756e69 636f692f 65746e6f 682e7478 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.050273] 3c90  6d6f682f 6b732f65 6f696479 69766e2f 5f616964 6e72656b 692f6c65 756c636e
May  2 14:11:21 deadfire3 kernel: [ 8676.058584] 3cb0  6c2f6564 78756e69 6572662f 72657a65 0000682e 00000000 6b736174 7274735f
May  2 14:11:21 deadfire3 kernel: [ 8676.066901] 3cd0  00746375 00000000 7465735f 6469745f 6464615f 73736572 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.075215] 3cf0  2d6d6d26 616d6d3e 65735f70 0000006d 3e2d7826 74696177 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.083537]
May  2 14:11:21 deadfire3 kernel: [ 8676.083537] X24: 0xffffffc0000d0e58:
May  2 14:11:21 deadfire3 kernel: [ 8676.088587] 0e58  910003e1 9272c424 f9400884 f900009f 942a7540 aa0003f5 91006264 f9400e65
May  2 14:11:21 deadfire3 kernel: [ 8676.096899] 0e78  eb05009f 540000e0 f9400483 f90004a3 f9000065 f9000e64 f9000484 14000009
May  2 14:11:21 deadfire3 kernel: [ 8676.105217] 0e98  91002280 f9400681 eb00003f 540000a0 aa1403e0 2a1703e1 aa1603e2 94002435
May  2 14:11:21 deadfire3 kernel: [ 8676.113529] 0eb8  aa1403e0 aa1503e1 942a75a6 a94153f3 a9425bf5 f9401bf7 a8c47bfd d65f03c0
May  2 14:11:21 deadfire3 kernel: [ 8676.121842] 0ed8  a9be7bfd 910003fd f9000bf3 aa0003f3 940045e3 34000100 91006261 f9401262
May  2 14:11:21 deadfire3 kernel: [ 8676.130150] 0ef8  f9400e63 f9000462 f9000043 f9000e61 f9000421 f9400bf3 a8c27bfd d65f03c0
May  2 14:11:21 deadfire3 kernel: [ 8676.138460] 0f18  a9bf7bfd 910003fd aa0003e4 f85f0005 f9400066 52800000 eb0600bf 54000201
May  2 14:11:21 deadfire3 kernel: [ 8676.146766] 0f38  b85f8086 b9400867 6b0700df 54000181 1100fcc0 6b1f00df 1a86b000 13067c00
May  2 14:11:21 deadfire3 kernel: [ 8676.155093]
May  2 14:11:21 deadfire3 kernel: [ 8676.155093] X25: 0xffffffc0d6313da0:
May  2 14:11:21 deadfire3 kernel: [ 8676.160147] 3da0  d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.168486] 3dc0  01003d00 ffffffc0 fc82dbc0 ffffffc0 00750460 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.176804] 3de0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.185115] 3e00  d19a7b50 ffffffc0 00000000 00000000 fe172fc0 ffffffc0 000d0ed8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.193427] 3e20  d6313e20 ffffffc0 d6313e20 ffffffc0 00000000 00000000 00085520 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.201735] 3e40  000d03f8 ffffffc0 d19a7b50 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.210050] 3e60  00000000 00000000 000dcc5c ffffffc0 d6313eb0 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.218364] 3e80  1fe585c0 ffffffc0 fc82dbc0 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.226676]
May  2 14:11:21 deadfire3 kernel: [ 8676.226676] X26: 0xffffffc0fc82e150:
May  2 14:11:21 deadfire3 kernel: [ 8676.231725] e150  00000000 00000000 00000000 00000000 00000005 00000005 00000005 00000005
May  2 14:11:21 deadfire3 kernel: [ 8676.240036] e170  00000004 ffffffc0 faff0400 ffffffc0 d3572800 ffffffc0 fb3e3800 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.248351] e190  00000014 00000000 00000002 fd96d6ac 0269bf88 00000000 fe3fb0c0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.256661] e1b0  fe172fc0 ffffffc0 f82df82d 00000000 fc82e1c0 ffffffc0 fc82e1c0 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.264971] e1d0  a1f7a1f7 00000000 fc82e1d8 ffffffc0 fc82e1d8 ffffffc0 00000001 00000017
May  2 14:11:21 deadfire3 kernel: [ 8676.273293] e1f0  00000000 00000000 fc82dfe8 ffffffc0 d51bb540 ffffffc0 00c315d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.281618] e210  00c2fa00 ffffffc0 fc82dbc0 ffffffc0 00000378 00002000 00000000 00000002
May  2 14:11:21 deadfire3 kernel: [ 8676.289932] e230  fab3c800 ffffffc0 faff0400 ffffffc0 d3572800 ffffffc0 fb3e3800 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.298251]
May  2 14:11:21 deadfire3 kernel: [ 8676.298251] X28: 0xffffffc0d630ff80:
May  2 14:11:21 deadfire3 kernel: [ 8676.303298] ff80  cfc4c4cf c2cfc4c2 c3c2cfc4 cbbfc1ce becbbfbe bfbecbbf cbbfbecb becbbfbe
May  2 14:11:21 deadfire3 kernel: [ 8676.311611] ffa0  c1bfccc1 cbbfbfcc becbbfbe c1becac1 c9bebeca bdc8bdbf bcbac7bc c8bfbac7
May  2 14:11:21 deadfire3 kernel: [ 8676.319925] ffc0  bbc8bfbb bfbdc7bf c6bbbdc7 b9c6bbb9 bab8c5ba c6bbb8c5 b8c5bab9 bab9c6bb
May  2 14:11:21 deadfire3 kernel: [ 8676.328234] ffe0  c5bab8c5 b9c6bbb8 b9b7c4b9 c3bab7c4 b8c1b9b9 bab9c3ba c3bab9c3 bac4bbb9
May  2 14:11:21 deadfire3 kernel: [ 8676.336550] 0000  00000002 00000000 ffffffff ffffffff fe172fc0 ffffffc0 000bce50 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.344860] 0020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.353174] 0040  00000000 00000000 00000002 00000002 00000100 00000000 57ac6e9d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.361500] 0060  3f37363d 343e3535 38364038 41393640 36403838 38364038 41393640 36403838
May  2 14:11:21 deadfire3 kernel: [ 8676.369811]
May  2 14:11:21 deadfire3 kernel: [ 8676.369811] X29: 0xffffffc0d6313cc0:
May  2 14:11:21 deadfire3 kernel: [ 8676.374859] 3cc0  00000001 00000000 00000001 00000000 00000005 00000000 01003c90 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.383182] 3ce0  000d0ed8 ffffffc0 d6313e20 ffffffc0 fc82e1d0 ffffffc0 00000001 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.391504] 3d00  d6310000 ffffffc0 d6313d40 ffffffc0 00750064 ffffffc0 d6313d40 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.399822] 3d20  0074fe10 ffffffc0 60000145 00000000 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.408149] 3d40  d6313d70 ffffffc0 00750064 ffffffc0 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.416461] 3d60  fc82e41c ffffffc0 014179f0 ffffffc0 d6313da0 ffffffc0 007505d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.424782] 3d80  fc82dbc0 ffffffc0 faff0400 ffffffc0 000021e3 00000000 171ca58d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.433094] 3da0  d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.441410]
May  2 14:11:21 deadfire3 kernel: [ 8676.442904] Process vi-output-4, tc (pid: 20545, stack limit = 0xffffffc0d6310058)
May  2 14:11:21 deadfire3 kernel: [ 8676.450467] Stack: (0xffffffc0d6313d40 to 0xffffffc0d6314000)
May  2 14:11:21 deadfire3 kernel: [ 8676.456212] 3d40: d6313d70 ffffffc0 00750064 ffffffc0 fc82dbc0 ffffffc0 faff0400 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.464382] 3d60: fc82e41c ffffffc0 014179f0 ffffffc0 d6313da0 ffffffc0 007505d8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.472553] 3d80: fc82dbc0 ffffffc0 faff0400 ffffffc0 000021e3 00000000 171ca58d 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.480723] 3da0: d6313e30 ffffffc0 000d04b8 ffffffc0 d19a7b50 ffffffc0 01415b28 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.488893] 3dc0: 01003d00 ffffffc0 fc82dbc0 ffffffc0 00750460 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.497062] 3de0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.505234] 3e00: d19a7b50 ffffffc0 00000000 00000000 fe172fc0 ffffffc0 000d0ed8 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.513406] 3e20: d6313e20 ffffffc0 d6313e20 ffffffc0 00000000 00000000 00085520 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.521576] 3e40: 000d03f8 ffffffc0 d19a7b50 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.529748] 3e60: 00000000 00000000 000dcc5c ffffffc0 d6313eb0 ffffffc0 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.537918] 3e80: 1fe585c0 ffffffc0 fc82dbc0 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.546088] 3ea0: d6313ea0 ffffffc0 d6313ea0 ffffffc0 00000000 00000000 00000000 ffffffc0
May  2 14:11:21 deadfire3 kernel: [ 8676.554258] 3ec0: d6313ec0 ffffffc0 d6313ec0 ffffffc0 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.562428] 3ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.570599] 3f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.578769] 3f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.586940] 3f40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.595110] 3f60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.603282] 3f80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.611450] 3fa0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.619620] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000005 00000000
May  2 14:11:21 deadfire3 kernel: [ 8676.627792] 3fe0: 00000000 00000000 00000000 00000000 565c4b45 7f797f5c a79a9499 797fa8a2
May  2 14:11:21 deadfire3 kernel: [ 8676.635960] Call trace:
May  2 14:11:21 deadfire3 kernel: [ 8676.638409] [<ffffffc00074fe10>] free_ring_buffers+0x3c/0xd0
May  2 14:11:21 deadfire3 kernel: [ 8676.644065] [<ffffffc000750064>] tegra_channel_release_frame+0xa0/0xac
May  2 14:11:21 deadfire3 kernel: [ 8676.650587] [<ffffffc0007505d8>] tegra_channel_kthread_release+0x178/0x1a4
May  2 14:11:21 deadfire3 kernel: [ 8676.657455] [<ffffffc0000d04b8>] kthread+0xc0/0xc8
May  2 14:11:21 deadfire3 kernel: [ 8676.662245] Code: f8607a60 b948d661 11000422 b908d662 (b9003801)
May  2 14:11:21 deadfire3 kernel: [ 8676.674126] ---[ end trace 7a9d03f4c84d921b ]---


## Userspace thread


May  4 09:44:30 deadfire3 kernel: [   60.123409] platform d.regulator: Driver reg-fixed-sync-voltage requests probe deferral
May  4 09:45:41 deadfire3 kernel: [  131.530657] tc358840 1-000f: enable_stream: disable
May  4 09:45:41 deadfire3 kernel: [  131.532778] tc358840 1-000f: tc358840_set_csi: Enabling CSI TX0
May  4 09:45:41 deadfire3 kernel: [  131.533126] tc358840 1-000f: tc358840_set_pll:
May  4 09:45:41 deadfire3 kernel: [  131.533132] tc358840 1-000f: tc358840_set_pll: Updating PLL clock of CSI TX0, hsck=549818136
May  4 09:45:41 deadfire3 kernel: [  131.550845] tc358840 1-000f: tc358840_set_csi: Disabling CSI TX1
May  4 09:45:44 deadfire3 kernel: [  134.602004] tc358840 1-000f: enable_stream: enable
May  4 09:45:44 deadfire3 kernel: [  134.604160] tc358840 1-000f: enable_stream: Stream enabled! Remaining timeout attempts: 100
May  4 09:45:44 deadfire3 kernel: [  134.811831] video4linux video3: frame start syncpt timeout!0
May  4 09:46:04 deadfire3 kernel: [  154.831841] video4linux video3: MW_ACK_DONE syncpoint time out!0
May  4 09:46:04 deadfire3 kernel: [  154.838520] Unable to handle kernel paging request at virtual address dead000000100108
May  4 09:46:04 deadfire3 kernel: [  154.848495] pgd = ffffffc0e0fbb000
May  4 09:46:04 deadfire3 kernel: [  154.853344] [dead000000100108] *pgd=0000000160fbc003, *pmd=0000000000000000
May  4 09:46:04 deadfire3 kernel: [  154.861295] Internal error: Oops: 96000044 [#1] PREEMPT SMP
May  4 09:46:04 deadfire3 kernel: [  154.866939] Enter nvdumper_crash_setup_regs
May  4 09:46:04 deadfire3 kernel: [  154.871340] nvdumper: all registers are saved.
May  4 09:46:04 deadfire3 kernel: [  154.871355] nvdumper: all registers are saved.
May  4 09:46:04 deadfire3 kernel: [  154.871369] nvdumper: all registers are saved.
May  4 09:46:04 deadfire3 kernel: [  154.884859] nvdumper: all registers are saved.
May  4 09:46:06 deadfire3 kernel: [  154.889358] Modules linked in: bcmdhd cfg80211 bluedroid_pm
May  4 09:46:06 deadfire3 kernel: [  154.895200] CPU: 0 PID: 1110 Comm: frontend_launch Not tainted 3.10.96-skydio-g9a5b6a4739e-dirty #151
May  4 09:46:06 deadfire3 kernel: [  154.904499] task: ffffffc0fd671000 ti: ffffffc0e0f60000 task.ti: ffffffc0e0f60000
May  4 09:46:06 deadfire3 kernel: [  154.912082] PC is at tegra_channel_queued_buf_done+0x6c/0x168
May  4 09:46:06 deadfire3 kernel: [  154.917911] LR is at tegra_channel_queued_buf_done+0x64/0x168
May  4 09:46:06 deadfire3 kernel: [  154.923725] pc : [<ffffffc000750060>] lr : [<ffffffc000750058>] pstate: 20000145
May  4 09:46:06 deadfire3 kernel: [  154.931188] sp : ffffffc0e0f63b50
May  4 09:46:06 deadfire3 kernel: [  154.934560] x29: ffffffc0e0f63b50 x28: 0000000000000001
May  4 09:46:06 deadfire3 kernel: [  154.940017] x27: ffffffc0fc016418 x26: ffffffc0fc015bc0
May  4 09:46:06 deadfire3 kernel: [  154.945461] x25: 0000000000000100 x24: 0000000000000200
May  4 09:46:06 deadfire3 kernel: [  154.950908] x23: ffffffc0fc0163f8 x22: ffffffc0fc016408
May  4 09:46:06 deadfire3 kernel: [  154.956348] x21: 0000000000000005 x20: dead0000000ffdb8
May  4 09:46:06 deadfire3 kernel: [  154.961798] x19: ffffffc0fa22d800 x18: 0000007fc334b1f4
May  4 09:46:06 deadfire3 kernel: [  154.967241] x17: 0000007f8e08b880 x16: 0000000000000000
May  4 09:46:06 deadfire3 kernel: [  154.972683] x15: 0000000000000000 x14: 0000000000000004
May  4 09:46:06 deadfire3 kernel: [  154.978124] x13: 0000000000000040 x12: 0000000000000002
May  4 09:46:06 deadfire3 kernel: [  154.983562] x11: ffffffc0e4941dc0 x10: ffffffc0e4941dc0
May  4 09:46:06 deadfire3 kernel: [  154.989005] x9 : ffffffc0e0f63880 x8 : ffffffc0fd671520
May  4 09:46:06 deadfire3 kernel: [  154.994448] x7 : ffffffc01fe58670 x6 : ffffffc001007098
May  4 09:46:06 deadfire3 kernel: [  154.999888] x5 : 0000000000000d3b x4 : 0000000000000000
May  4 09:46:06 deadfire3 kernel: [  155.005330] x3 : 0000000000000000 x2 : dead000000100100
May  4 09:46:06 deadfire3 kernel: [  155.010772] x1 : 0000000000000005 x0 : dead000000200200
May  4 09:46:06 deadfire3 kernel: [  155.016221]
May  4 09:46:06 deadfire3 kernel: [  155.016221] PC: 0xffffffc00074ffe0:
May  4 09:46:06 deadfire3 kernel: [  155.021255] ffe0  52800021 97ffff5d a94153f3 a8c37bfd d65f03c0 a9ba7bfd 910003fd a90153f3
May  4 09:46:06 deadfire3 kernel: [  155.029837] 0000  a9025bf5 a90363f7 a9046bf9 f9002bfb aa0003fa 2a0103f5 9120e017 91212016
May  4 09:46:06 deadfire3 kernel: [  155.038407] 0020  9121601b aa1b03e0 9410778e f9441f40 d10d2013 f9400014 d10d2294 eb0002ff
May  4 09:46:06 deadfire3 kernel: [  155.046986] 0040  54000340 d2802019 d2804018 aa1303e0 2a1503e1 97ffb642 f941aa60 f941a662
May  4 09:46:06 deadfire3 kernel: [  155.055559] 0060  f9000440 f9000002 aa1903e0 f2a00200 f2fbd5a0 f901a660 aa1803e2 f2a00402
May  4 09:46:06 deadfire3 kernel: [  155.064127] 0080  f2fbd5a2 f901aa62 f941a680 d10d2001 910d2280 eb0002ff 54000080 aa1403f3
May  4 09:46:06 deadfire3 kernel: [  155.072701] 00a0  aa0103f4 17ffffea aa1b03e0 941077bc 91217358 aa1803e0 9410776a f9442740
May  4 09:46:06 deadfire3 kernel: [  155.081273] 00c0  d10d2013 f9400014 d10d2294 eb0002df 54000340 d2802019 d2804017 aa1303e0
May  4 09:46:06 deadfire3 kernel: [  155.089846]
May  4 09:46:06 deadfire3 kernel: [  155.089846] LR: 0xffffffc00074ffd8:
May  4 09:46:06 deadfire3 kernel: [  155.094880] ffd8  b8207a61 aa1303e0 52800021 97ffff5d a94153f3 a8c37bfd d65f03c0 a9ba7bfd
May  4 09:46:06 deadfire3 kernel: [  155.103462] fff8  910003fd a90153f3 a9025bf5 a90363f7 a9046bf9 f9002bfb aa0003fa 2a0103f5
May  4 09:46:06 deadfire3 kernel: [  155.112038] 0018  9120e017 91212016 9121601b aa1b03e0 9410778e f9441f40 d10d2013 f9400014
May  4 09:46:06 deadfire3 kernel: [  155.120618] 0038  d10d2294 eb0002ff 54000340 d2802019 d2804018 aa1303e0 2a1503e1 97ffb642
May  4 09:46:06 deadfire3 kernel: [  155.129188] 0058  f941aa60 f941a662 f9000440 f9000002 aa1903e0 f2a00200 f2fbd5a0 f901a660
May  4 09:46:06 deadfire3 kernel: [  155.137767] 0078  aa1803e2 f2a00402 f2fbd5a2 f901aa62 f941a680 d10d2001 910d2280 eb0002ff
May  4 09:46:06 deadfire3 kernel: [  155.146329] 0098  54000080 aa1403f3 aa0103f4 17ffffea aa1b03e0 941077bc 91217358 aa1803e0
May  4 09:46:06 deadfire3 kernel: [  155.154905] 00b8  9410776a f9442740 d10d2013 f9400014 d10d2294 eb0002df 54000340 d2802019
May  4 09:46:06 deadfire3 kernel: [  155.163481]
May  4 09:46:06 deadfire3 kernel: [  155.163481] SP: 0xffffffc0e0f63ad0:
May  4 09:46:06 deadfire3 kernel: [  155.168514] 3ad0  000ffdb8 dead0000 00000005 00000000 fc016408 ffffffc0 fc0163f8 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.177084] 3af0  00000200 00000000 00000100 00000000 fc015bc0 ffffffc0 fc016418 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.185660] 3b10  00000001 00000000 e0f63b50 ffffffc0 00750058 ffffffc0 e0f63b50 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.194251] 3b30  00750060 ffffffc0 20000145 00000000 00000005 00000000 fc016408 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.202827] 3b50  e0f63bb0 ffffffc0 00750370 ffffffc0 fc015bc0 ffffffc0 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  155.211390] 3b70  fc0161e8 ffffffc0 dc99b9c0 ffffffc0 00c31238 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.219963] 3b90  00000000 00000000 e0f63d70 ffffffc0 dc9f21c0 ffffffc0 dc99b9c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.228551] 3bb0  e0f63bd0 ffffffc0 0073d5c4 ffffffc0 fc0161e8 ffffffc0 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  155.237149]
May  4 09:46:06 deadfire3 kernel: [  155.237149] X6: 0xffffffc001007018:
May  4 09:46:06 deadfire3 kernel: [  155.242185] 7018  656e203e 635f7478 3d6d6d6f 6e207325 5f747865 3d646970 6e206425 5f747865
May  4 09:46:06 deadfire3 kernel: [  155.250760] 7038  6f697270 0a64253d 00000000 00000000 6d6f682f 6b732f65 6f696479 69766e2f
May  4 09:46:06 deadfire3 kernel: [  155.259339] 7058  5f616964 6e72656b 692f6c65 756c636e 742f6564 65636172 6576652f 2f73746e
May  4 09:46:06 deadfire3 kernel: [  155.267919] 7078  65686373 00682e64 65686373 65665f64 72757461 00007365 005f4f4e 00000000
May  4 09:46:06 deadfire3 kernel: [  155.276510] 7098  6d6f682f 6b732f65 6f696479 69766e2f 5f616964 6e72656b 6b2f6c65 656e7265
May  4 09:46:06 deadfire3 kernel: [  155.285092] 70b8  63732f6c 2f646568 65726f63 0000632e 55424544 4f4c5f47 5f534b43 4e524157
May  4 09:46:06 deadfire3 kernel: [  155.293675] 70d8  284e4f5f 00297325 65727028 74706d65 756f635f 2928746e 30203c20 00000029
May  4 09:46:06 deadfire3 kernel: [  155.302251] 70f8  65727028 74706d65 756f635f 2928746e 50202620 4d454552 4d5f5450 294b5341
May  4 09:46:06 deadfire3 kernel: [  155.310836]
May  4 09:46:06 deadfire3 kernel: [  155.310836] X7: 0xffffffc01fe585f0:
May  4 09:46:06 deadfire3 kernel: [  155.315867] 85f0  ffffc74c 00000000 00000000 00000000 00000000 00000000 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  155.324447] 8610  0d927c65 00000024 473ab000 00006eab 0008f7f6 00000000 00000c00 00000000
May  4 09:46:06 deadfire3 kernel: [  155.333023] 8630  00000000 00000000 00003bb1 00000000 0003e3ab 00000000 00000c00 00000000
May  4 09:46:06 deadfire3 kernel: [  155.341599] 8650  00000000 00000000 00000003 00000003 055135ba 00000009 961304c7 00000006
May  4 09:46:06 deadfire3 kernel: [  155.350181] 8670  fd5d8e98 ffffffc0 ffde6c18 ffffffc0 fd671048 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.358764] 8690  00000000 00000000 00000000 00000000 0000023d 00000000 00000138 00000000
May  4 09:46:06 deadfire3 kernel: [  155.367334] 86b0  00000059 00000000 000240da 00000000 00000000 00000000 000240d9 00000000
May  4 09:46:06 deadfire3 kernel: [  155.375915] 86d0  000001ac 00000000 00000191 00000000 00000c00 00000000 1fe585c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.384507]
May  4 09:46:06 deadfire3 kernel: [  155.384507] X8: 0xffffffc0fd6714a0:
May  4 09:46:06 deadfire3 kernel: [  155.389535] 14a0  75616c5f 0068636e 00000000 00000000 ef048640 ffffffc0 0000bc59 00000000
May  4 09:46:06 deadfire3 kernel: [  155.398120] 14c0  fffa6180 ffffffc0 fd671000 ffffffc0 fffa6180 ffffffc0 012005c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.406709] 14e0  00000000 00000000 012005c0 ffffffc0 012005c0 ffffffc0 013b04a0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.415281] 1500  fb2903c0 ffffffc0 01460168 ffffffc0 e0f63880 ffffffc0 e0f63880 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.423848] 1520  00086ae0 ffffffc0 9ad796f0 0000007f 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.432424] 1540  00000000 40240000 00000000 00000000 00000000 40240000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.440998] 1560  00000000 00000000 00000000 40026000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.449572] 1580  00000001 00000000 00000001 00000000 00000000 40000000 00000000 40000000
May  4 09:46:06 deadfire3 kernel: [  155.458155]
May  4 09:46:06 deadfire3 kernel: [  155.458155] X9: 0xffffffc0e0f63800:
May  4 09:46:06 deadfire3 kernel: [  155.463182] 3800  fc016418 ffffffc0 00000001 00000000 00ffef58 ffffffc0 00ffef68 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.471777] 3820  00000000 00000000 00324d08 00324d08 e0f63890 ffffffc0 000868c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.480361] 3840  ffffffff 00000000 e0f63a30 ffffffc0 00fff020 ffffffc0 0105ed28 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.488948] 3860  fd671000 ffffffc0 00000025 00000000 00000100 00000000 00390084 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.497538] 3880  00324d18 ffffffc0 00003958 ffffffc0 e0f638c0 ffffffc0 00089fc4 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.506118] 38a0  00000001 00000000 01001468 ffffffc0 e0f63a30 ffffffc0 e0f60000 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.514704] 38c0  e0f63900 ffffffc0 0008a0dc ffffffc0 e0f63a30 ffffffc0 e0f60000 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.523290] 38e0  96000044 00000000 01001468 ffffffc0 20000145 00000000 0008a0c8 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.531874]
May  4 09:46:06 deadfire3 kernel: [  155.531874] X10: 0xffffffc0e4941d40:
May  4 09:46:06 deadfire3 kernel: [  155.536991] 1d40  00000002 00000000 e4941d48 ffffffc0 e4941d48 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.545572] 1d60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.554143] 1d80  0126d148 ffffffc0 fa24a400 ffffffc0 e4941d88 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.562716] 1da0  e48cbde0 ffffffc0 00000020 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.571285] 1dc0  ffffffff 0000ffff 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.579853] 1de0  42000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.588419] 1e00  00100100 dead0000 00200200 dead0000 00000000 00000000 d0b3c000 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.596990] 1e20  00000000 00000000 00000000 00000000 ffffffff 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.605574]
May  4 09:46:06 deadfire3 kernel: [  155.605574] X11: 0xffffffc0e4941d40:
May  4 09:46:06 deadfire3 kernel: [  155.610692] 1d40  00000002 00000000 e4941d48 ffffffc0 e4941d48 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.619269] 1d60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.627841] 1d80  0126d148 ffffffc0 fa24a400 ffffffc0 e4941d88 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.636428] 1da0  e48cbde0 ffffffc0 00000020 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.645001] 1dc0  ffffffff 0000ffff 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.653582] 1de0  42000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.662161] 1e00  00100100 dead0000 00200200 dead0000 00000000 00000000 d0b3c000 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.670733] 1e20  00000000 00000000 00000000 00000000 ffffffff 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.679323]
May  4 09:46:06 deadfire3 kernel: [  155.679323] X19: 0xffffffc0fa22d780:
May  4 09:46:06 deadfire3 kernel: [  155.684440] d780  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.693015] d7a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.701577] d7c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.710146] d7e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.718711] d800  00000001 00000001 00000000 00000000 00000001 00000000 d5be3da0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.727270] d820  ef9dd023 ffffffff 00000000 00000000 00000000 00000000 00000253 00000002
May  4 09:46:06 deadfire3 kernel: [  155.735843] d840  00000000 00000000 00000000 00000000 00000000 00000000 001c2000 001c2000
May  4 09:46:06 deadfire3 kernel: [  155.744417] d860  9991d000 0000007f e0f63ce8 ffffffc0 01014f70 ffffffc0 e0f63dd0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.753002]
May  4 09:46:06 deadfire3 kernel: [  155.753002] X22: 0xffffffc0fc016388:
May  4 09:46:06 deadfire3 kernel: [  155.758116] 6388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.766707] 63a8  00000000 00000000 00000000 00000000 00000000 00000000 001c2000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.775278] 63c8  00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  155.783863] 63e8  00000000 00000000 fc001cc0 ffffffc0 fa22db48 ffffffc0 fa22db48 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.792433] 6408  fc016408 ffffffc0 fc016408 ffffffc0 36e836e7 025d025d 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.800998] 6428  00040004 00000000 fc016430 ffffffc0 fc016430 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.809564] 6448  00040004 00000000 fc016450 ffffffc0 fc016450 ffffffc0 02980500 ffffff80
May  4 09:46:06 deadfire3 kernel: [  155.818136] 6468  00000000 00000000 00000000 00000000 00000040 00000001 00000001 00000001
May  4 09:46:06 deadfire3 kernel: [  155.826714]
May  4 09:46:06 deadfire3 kernel: [  155.826714] X23: 0xffffffc0fc016378:
May  4 09:46:06 deadfire3 kernel: [  155.831830] 6378  fc016370 ffffffc0 fc001cc0 ffffffc0 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.840410] 6398  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.848976] 63b8  00000000 00000000 001c2000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.857560] 63d8  00000000 00000000 00000001 00000000 00000000 00000000 fc001cc0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.866138] 63f8  fa22db48 ffffffc0 fa22db48 ffffffc0 fc016408 ffffffc0 fc016408 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.874703] 6418  36e836e7 025d025d 00000000 00000000 00040004 00000000 fc016430 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.883280] 6438  fc016430 ffffffc0 00000000 00000000 00040004 00000000 fc016450 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.891845] 6458  fc016450 ffffffc0 02980500 ffffff80 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.900425]
May  4 09:46:06 deadfire3 kernel: [  155.900425] X26: 0xffffffc0fc015b40:
May  4 09:46:06 deadfire3 kernel: [  155.905541] 5b40  00000040 00000001 00000001 00000001 00000002 00000000 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  155.914115] 5b60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.922680] 5b80  00008119 00000000 00000000 00000000 fc33d740 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.931261] 5ba0  00000000 00000000 00000000 00000000 00000004 00000000 012a83f0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.939819] 5bc0  00000000 00000000 00000000 00000000 fd5cd958 ffffffc0 fc015298 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.948391] 5be0  fc0120e8 ffffffc0 00000004 00000000 fc015f20 ffffffc0 00010001 00000000
May  4 09:46:06 deadfire3 kernel: [  155.956960] 5c00  00000000 00000000 00000000 00010001 00010001 00000000 fc015fd0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  155.965541] 5c20  fc001d40 ffffffc0 00000000 00000000 00000001 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.974130]
May  4 09:46:06 deadfire3 kernel: [  155.974130] X27: 0xffffffc0fc016398:
May  4 09:46:06 deadfire3 kernel: [  155.979245] 6398  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.987817] 63b8  00000000 00000000 001c2000 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  155.996395] 63d8  00000000 00000000 00000001 00000000 00000000 00000000 fc001cc0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.004968] 63f8  fa22db48 ffffffc0 fa22db48 ffffffc0 fc016408 ffffffc0 fc016408 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.013536] 6418  36e836e7 025d025d 00000000 00000000 00040004 00000000 fc016430 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.022111] 6438  fc016430 ffffffc0 00000000 00000000 00040004 00000000 fc016450 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.030679] 6458  fc016450 ffffffc0 02980500 ffffff80 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.039260] 6478  00000040 00000001 00000001 00000001 00000004 00000000 00000001 00000254
May  4 09:46:06 deadfire3 kernel: [  156.047842]
May  4 09:46:06 deadfire3 kernel: [  156.047842] X29: 0xffffffc0e0f63ad0:
May  4 09:46:06 deadfire3 kernel: [  156.052960] 3ad0  000ffdb8 dead0000 00000005 00000000 fc016408 ffffffc0 fc0163f8 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.061554] 3af0  00000200 00000000 00000100 00000000 fc015bc0 ffffffc0 fc016418 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.070133] 3b10  00000001 00000000 e0f63b50 ffffffc0 00750058 ffffffc0 e0f63b50 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.078715] 3b30  00750060 ffffffc0 20000145 00000000 00000005 00000000 fc016408 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.087283] 3b50  e0f63bb0 ffffffc0 00750370 ffffffc0 fc015bc0 ffffffc0 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  156.095893] 3b70  fc0161e8 ffffffc0 dc99b9c0 ffffffc0 00c31238 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.104465] 3b90  00000000 00000000 e0f63d70 ffffffc0 dc9f21c0 ffffffc0 dc99b9c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.113042] 3bb0  e0f63bd0 ffffffc0 0073d5c4 ffffffc0 fc0161e8 ffffffc0 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  156.121608]
May  4 09:46:06 deadfire3 kernel: [  156.123162] Process frontend_launch (pid: 1110, stack limit = 0xffffffc0e0f60058)
May  4 09:46:06 deadfire3 kernel: [  156.130723] Stack: (0xffffffc0e0f63b50 to 0xffffffc0e0f64000)
May  4 09:46:06 deadfire3 kernel: [  156.136546] 3b40:                                     e0f63bb0 ffffffc0 00750370 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.144814] 3b60: fc015bc0 ffffffc0 00000001 00000000 fc0161e8 ffffffc0 dc99b9c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.153077] 3b80: 00c31238 ffffffc0 00000000 00000000 00000000 00000000 e0f63d70 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.161343] 3ba0: dc9f21c0 ffffffc0 dc99b9c0 ffffffc0 e0f63bd0 ffffffc0 0073d5c4 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.169605] 3bc0: fc0161e8 ffffffc0 00000001 00000000 e0f63c00 ffffffc0 0073d848 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.177870] 3be0: fc0161e8 ffffffc0 00000001 00000000 012a72d0 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.186133] 3c00: e0f63c20 ffffffc0 0073f028 ffffffc0 dc99b9c0 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.194396] 3c20: e0f63c40 ffffffc0 00729bb4 ffffffc0 fc015bd0 ffffffc0 40045613 00000000
May  4 09:46:06 deadfire3 kernel: [  156.202657] 3c40: e0f63c50 ffffffc0 0072dda4 ffffffc0 e0f63ce0 ffffffc0 0072db10 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.210920] 3c60: 00000000 00000000 40045613 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.219184] 3c80: e0f63d70 ffffffc0 00000001 00000000 dcb13220 0000007f 00000004 00000000
May  4 09:46:06 deadfire3 kernel: [  156.227447] 3ca0: 00000004 00000000 dc99b9c0 ffffffc0 dc9f21c0 ffffffc0 dcb12f18 00000001
May  4 09:46:06 deadfire3 kernel: [  156.235710] 3cc0: 00000009 00000000 00000015 00000000 00000116 00000000 000000d3 00000000
May  4 09:46:06 deadfire3 kernel: [  156.243972] 3ce0: e0f63df0 ffffffc0 0072dc28 ffffffc0 fc015bd0 ffffffc0 fc015fe8 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.252234] 3d00: 40045613 00000000 ffffffed 00000000 dcb13220 0000007f dc99b9c0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.260499] 3d20: 00000116 00000000 0000001d 00000000 01228000 ffffffc0 e0f60000 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.268760] 3d40: 00518b00 00000000 0072dc30 ffffffc0 e0f63da0 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.277022] 3d60: 00000000 00000000 00000000 00000000 00000001 ffffffc0 00000002 00000000
May  4 09:46:06 deadfire3 kernel: [  156.285283] 3d80: faf535d0 ffffffc0 00000001 00000000 faf535c0 ffffffc0 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.293548] 3da0: 2d1f0002 434cffef 00000000 00000000 e041f280 ffffffc0 00000005 00000000
May  4 09:46:06 deadfire3 kernel: [  156.301813] 3dc0: e0f63df0 ffffffc0 009ab6a8 ffffffc0 e0f63de0 ffffffc0 00b6b5f0 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.310078] 3de0: e0f63e00 ffffffc0 0072897c ffffffc0 e0f63e00 ffffffc0 007289a4 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.318342] 3e00: e0f63e40 ffffffc0 001be4c4 ffffffc0 dcb13220 0000007f fd6d2380 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.326608] 3e20: 00000018 00000000 dcb13220 0000007f 40045613 00000000 00000001 00000000
May  4 09:46:06 deadfire3 kernel: [  156.334872] 3e40: e0f63e50 ffffffc0 001bf194 ffffffc0 e0f63e80 ffffffc0 001bf274 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.343132] 3e60: dc99b9c0 ffffffc0 00000000 00000000 e0f63e80 ffffffc0 001bf258 ffffffc0
May  4 09:46:06 deadfire3 kernel: [  156.351390] 3e80: dcb13160 0000007f 000853c8 ffffffc0 dcb13230 0000007f dcb13230 0000007f
May  4 09:46:06 deadfire3 kernel: [  156.359649] 3ea0: ffffffff ffffffff 9b2c210c 0000007f 20000000 00000000 00000015 00000000
May  4 09:46:06 deadfire3 kernel: [  156.367908] 3ec0: 80000000 00000000 00001148 00000001 00000018 00000000 40045613 00000000
May  4 09:46:06 deadfire3 kernel: [  156.376168] 3ee0: dcb13220 0000007f dcb13220 0000007f dcb13108 0000007f 00000002 00000000
May  4 09:46:06 deadfire3 kernel: [  156.384427] 3f00: 004a6d08 00000000 49444956 535f434f 0000001d 00000000 fefefeff fefefefe
May  4 09:46:06 deadfire3 kernel: [  156.392686] 3f20: 7f7f7f7f 7f7f7f7f 01010101 01010101 00000000 00000000 e8000000 00000003
May  4 09:46:06 deadfire3 kernel: [  156.400945] 3f40: 00000000 00000000 9b346588 0000007f 9a50b310 0000007f 9b2c2100 0000007f
May  4 09:46:06 deadfire3 kernel: [  156.409205] 3f60: dcb123f8 0000007f dcb13230 0000007f dcb13230 0000007f 00422610 00000000
May  4 09:46:06 deadfire3 kernel: [  156.417464] 3f80: dcb13300 0000007f dcb13c50 0000007f 9a530000 0000007f dcb13320 0000007f
May  4 09:46:06 deadfire3 kernel: [  156.425724] 3fa0: 0040f3b0 00000000 0040f3e0 00000000 dcb13310 0000007f dcb13160 0000007f
May  4 09:46:06 deadfire3 kernel: [  156.433981] 3fc0: 9a4ebaec 0000007f dcb13160 0000007f 9b2c210c 0000007f 20000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.442241] 3fe0: 00000018 00000000 0000001d 00000000 00000000 00000000 00000000 00000000
May  4 09:46:06 deadfire3 kernel: [  156.450479] Call trace:
May  4 09:46:06 deadfire3 kernel: [  156.453001] [<ffffffc000750060>] tegra_channel_queued_buf_done+0x6c/0x168
May  4 09:46:06 deadfire3 kernel: [  156.459866] [<ffffffc000750370>] tegra_channel_stop_streaming+0x54/0xc8
May  4 09:46:06 deadfire3 kernel: [  156.466560] [<ffffffc00073d5c4>] __vb2_queue_cancel+0x30/0xa8
May  4 09:46:06 deadfire3 kernel: [  156.472380] [<ffffffc00073d848>] vb2_streamoff+0xac/0x124
May  4 09:46:06 deadfire3 kernel: [  156.477846] [<ffffffc00073f028>] vb2_ioctl_streamoff+0x44/0x50
May  4 09:46:06 deadfire3 kernel: [  156.483751] [<ffffffc000729bb4>] v4l_streamoff+0x1c/0x24
May  4 09:46:06 deadfire3 kernel: [  156.489129] [<ffffffc00072dda4>] __video_do_ioctl+0x174/0x284
May  4 09:46:06 deadfire3 kernel: [  156.494940] [<ffffffc00072db10>] video_usercopy+0x1e4/0x2e8
May  4 09:46:06 deadfire3 kernel: [  156.500576] [<ffffffc00072dc28>] video_ioctl2+0x14/0x1c
May  4 09:46:06 deadfire3 kernel: [  156.505870] [<ffffffc0007289a4>] v4l2_ioctl+0x70/0x134
May  4 09:46:06 deadfire3 kernel: [  156.511082] [<ffffffc0001be4c4>] vfs_ioctl+0x1c/0x44
May  4 09:46:06 deadfire3 kernel: [  156.516120] [<ffffffc0001bf194>] do_vfs_ioctl+0x220/0x22c
May  4 09:46:06 deadfire3 kernel: [  156.521598] [<ffffffc0001bf274>] SyS_ioctl+0xd4/0x17c
May  4 09:46:06 deadfire3 kernel: [  156.526723] Code: 2a1503e1 97ffb642 f941aa60 f941a662 (f9000440)
May  4 09:46:06 deadfire3 kernel: [  156.539082] ---[ end trace 1cec99130188edfd ]---

Streaming Sound Between Devices


Today I spent a little time figuring out solutions for "casting" audio from one device to another. Here are some notes on how to make this happen.

Background

I have an old laptop setup as a media-center PC connected to my TV and receiver. This is pretty much the only set of speakers we have in the apartment so whenever we say "I'd like to listen to some music" we almost always mean "through these speakers". However we don't really have a good source of music these days. Sometimes we listen to youtube, sometimes amazon music, sometimes google music. Sometimes we play audio files from a laptop or desktop.

Ideally, I'd figure out a good media management solution, but until that day a decent stopgap and otherwise generally useful feature would be "play this music here on this device I'm using but make the sound come out of those nice speakers over there".

I'm currently able to listen to Google Play Music or Youtube from Firefox or Chrome on either Ubuntu system in the house (desktop or laptop) or from Google Play music or Amazon Prime Music from my Android phone. Amazon Prime Music is apparently broken (no audio) on Ubuntu period...

In theory this solution should work for any ubuntu application and any app on Android.

Other Options

I've previously tried or briefly entertained the following solutions:

Pulse Audio

My current solution is to "enable network access to local sound devices" on the Pulse Audio server running on the media center laptop connected to the receiver. The solution came from askubuntu.com but I'll describe it here as well.

Install paprefs

First, install the pulse audio preferences tool on the sound server:

$ sudo apt-get install paprefs

Configure pulse-audio

Run paprefs uncheck all of the options on all tabs except for

  • "Enable network access to local sound devices"
  • "Don't require authentication"

Restart pulse

Restart the pulse audio server with:

$ pulseaudio -k

Test the client

Test the client by downloading a sample audio file and playing it through the network to your audio server:

$ wget http://www.signalogic.com/melp/EngSamples/Orig/female.wav
$ paplay --server=192.168.x.x:4713 female.wav

Use the ip address of the actual server. If you don't know it, you can right click on the "Network Manager" applet and click "Connection Information" or you can run (in a terminal) ifconfig and read through the output.

Stream browser audio

Run your browser in an environment with PULSE_SERVER=192.168.x.x:4713 and give youtube a try. This worked for me on ubuntu 16.04 (on both client and server) and with chrome and firefox. Amazon prime streaming however doesn't seem to work.

Stream android audio

Solutions came from a couple places:

On the server:

$ sudo apt install pulseaudio-module-bluetooth
$ pactl load-module module-bluetooth-discover

Connect the phone and PC using bluetooth. For me, I had to initialize the pairing from the PC, rather than the phone.

In android bluetooth settings for the PC connection, click Use For: "Media Audio" (this was already checked for me).

At this point it should just work. If you go to the unity-control-center "Sound" section (or "Sound" on the dash menu) you should see the phone listed under the "Input" tab.

python-like kwargs in C++


When dealing with functions that have lots of optional parameters, or at least for which resonable defaults are readily available, it's often a bit of a frustration in C++. Generally defaults are specified during a function declaration as in:

double BinarySearch(std::function fn,
                    int max_depth = 16, double epsilon = 1e-9,
                    double lower_bound = 0, double upper_bound = 100);

And then if we call BinarySearch with only one parameter then the call will use the default values for the rest. But what if I want to specify custom bound, and use the defaults for the other parameters? Admittedly, this is a somewhat contrived example since bounds are less likely to be optional then the others, and we could reorder them better going from most-likely-to-be- specified to least, but it's easy to see how something more flexible would be desirable.

Consider then the following two code snippets. Which is more readable?

First snippet:

double solution = BinarySearch(fn, 0, 100);

Second snippet:

double solution = BinarySearch(fn, lower_bound = 0, upper_bound = 100);

I really like the way that optional arguments work in python with kwargs. I'd love to have that same kind of functionality in C++. kwargs.h implements one mechanism of achieving this.

How does it work

kwargs takes advantage of variadic templates in C++ to build up a single data structure which contains all of the optional parameters (I'll call this a "parameter pack"). Each optional parameter of type T is stored in a structure of type Arg<tag,T> where tag is a unique numeric identifier associated with a particular optional argument key.

The parameter pack data structure derives from Arg<tag,T> for each (tag,T) pair that shows up in the list of optional arguments.

Overloading of the assignment (=) operator gives us an opportunity for building the (tag,T) pairs within the parameter list of the function call.

Get it

See the source and doxygen on github.

Example

I'll conclude with an example usage:

#include 
#include "kwargs.h"

// these are tags which will uniquely identify the arguments in a parameter
// pack
enum Keys {
  c_tag,
  d_tag
};

// global symbols used as keys in list of kwargs
kw::Key c_key;
kw::Key d_key;

// a function taking kwargs parameter pack
template 
void foo(int a, int b, Args kwargs) {
  // first, we construct the parameter pack from the parameter pack
  kw::ParamPack params(kwargs);

  std::cout << "foo:\n--------"
            << "\na: " << a
            << "\nb: " << b
  // We can attempt to retrieve a key while providing a default fallback value.
  // If c_key is in kwargs then this will return the value associated with
  // that key, and will have the correct type. Note that the type of the default
  // parameter in this case is const char*.
            << "\nc: " << kw::Get(params,c_key,"null");
  // We can also do stuff conditionally based on whether or not arg exists in
  // the param pack. We still need to provide a default value, since we need to
  // know the return type of the Get function when the key is not in kwargs.
  // In this case, the default value wont ever be used at runtime.
  if( kw::ContainsTag::result ) {
    std::cout << "\nd: " << kw::Get(params,d_key,0);
  }

  std::cout << "\n\n";
}

int main( int argc, char** argv )
{
  foo(1, 2);
  foo(1, 2, c_key=3);
  foo(1, 2, c_key=3, d_key=4);
  foo(1, 2, d_key=4);
}

Full Disk Encryption with USB master key


When I decided to go with full disk encryption on my machines, I had a pretty hard time figuring out exactly what to do. Not only were there tools and commands to learn, but there was quite a bit of design involved in the process. In this post, I will describe the setup I chose, and the scripts I use to make things a bit more convenient.

Partition Design

partition size label file system mount point notes
sda1 200MiB boot ext4 /boot
sda2 20GiB luks/ext4 /
sda3 218GiB luks/ext4 /home
sda4 6GiB Extended Partition
sda5 4096MiB recovery ext4 / Minimal Ubuntu for recovery
sda6 1900MiB luks/swap

Creating the partition layout

After deciding on a partition scheme, the next step was to create the partitions and format them. I created a 14.04 USB installer and booted the machine in the "Try Ubuntu" mode.

This part is easy. I used gparted to create the partition layout above. For the three encrypted partitions (marked luks/… above) I left them "unformatted".

Formatting the encrypted partitions

Instructions for the next step came mostly from the Arch wiki. The process involves first formatting the encrypted partitions with a Linux Unified Key Setup (LUKS) header. The luks header contains the AES master key as well as key information for up to eight passphrases (note, in this context a passphrase may be a file of arbitrary data) allowing access to that master key.

Initially, I chose to lock down the root and home partitions using a keyfile passphrase, and later added a password as a passphrase, so I will go through that process.

First, we need to create a keyfile for each partition. I used /dev/random to source random bytes (rather than /dev/urandom which is faster but pseudorandom). Knowing that /dev/random sources user interface event arrival times for randomness I made sure to do lots of typing while I was reading off a key. I chose to use a keysize of 512bytes. I initially accomplished this with

$ cat /dev/random > root.key

while watching in another terminal with ls -l until the file was over 512 bytes. Then I used

$ truncate -s 512 root.key

to reduce the size to 512. After doing this a few times I created a small script to provide saner feedback on how much data had been read from /dev/random

#!/usr/bin/python
# key_progress.py
import sys

max = 512
bytes = 0
while( bytes < max ):
  data = sys.stdin.read(1)
  bytes += len(data)
  fraction = bytes/float(max)
  digits = int(10*fraction)
  bars = '='*digits
  spaces = ' '*(10-digits)
  percent = 100*fraction
  text = "\r[%s%s] : %04.2f%%" % (bars,spaces,percent)
  sys.stdout.write(text)
  sys.stdout.flush()
sys.stdout.write('\n')

Usage is like

$ cat /dev/random | tee -a root.key | key_progress.py

Next, I set up the root partition. As described in the wiki, we create the luks header with

# cryptsetup -cipher=aes-xts-plain64 -key-file=~/root.key -key-size=512 -hash=sha51 -iter-time=5000 -use-random luksFormat /dev/sda2

Please research the different cipher options but at the time of this writing my research indicates that aes-xts is generally the best option for a general- use large block device (hard drive).

Note that one may omit the -key-file=/path/to/file flag to specify a password instead of a keyfile, but I chose to start with a keyfile and add a password later.

Once the block device has a proper luks header we can expose the encrypted block device as an unencrypted block device with the following command.

# cryptsetup luksOpen /dev/sda2 root

This will expose the unencrypted blocks of /dev/sda2 in /dev/mapper/root as if it were a raw block device. We can then create an ext4 filesystem on that encrypted device via

# mkfs.ext4 /dev/mapper/root

I repeated the same steps for the home partition:

# cat /dev/random | tee -a home.key | key_progress.py
# cryptsetup -cipher=aes-xts-plain64 -key-file=~/home.key -key-size=512 -hash=sha51 -iter-time=5000 -use-random luksFormat /dev/sda3
# cryptsetup luksOpen /dev/sda3 home
# mkfs.ext4 /dev/mapper/home

Installing Ubuntu 14.04

The installation process is another of the easy steps. Just launch "install ubuntu" from the desktop icon. When it asks where to install choose "do something else" and specify the partitions manually.

device type mount point format?
/dev/sda1 ext4 /boot No
/dev/mapper/root ext4 / No
/dev/mapper/home ext4 /home No
/dev/sda6 swap

The rest of the installation is business as usual

Post-install work

After installing, and before rebooting we need to do some work on the new system.

First, let's setup an encrypted swap. If the ubuntu live-usb system happens to be using our formatted swap space then we need to turn off the swap. Then we run the following

# swapoff -a
# cryptsetup -d /dev/random create cryptswap /dev/sda6
# mkswap -f /dev/mapper/cryptswap -v1

At this point I realized that I had no way to provide the keyfile for unlocking the root partition at boot so I added a password to encrypted root partition.

# cryptsetup -key-file=/root.key luksAddKey /dev/sda2

Now we need to setup the new systems cryptab. Start by mounting the filesystem for the new installation.

# cd /mnt
# mkdir root
# mount /dev/mapper/root root
# mount /dev/mapper/home root/home
# mount /dev/sda1 root/boot

Now we edit /etc/cryptab to setup the encrypted volumes. My crypttab looks like the following:

# root filesystem
root UUID=fdc69d8e-ede4-40f1-8a38-d69ba73c1c82 none luks

# home filesystem
home UUID=74c4c574-2396-434e-9f0f-ea9f8270c44e /root/home.key luks

# swap
cryptswap /dev/sda6 /dev/urandom swap

You can find the UUID of the relevant partitions by running blkid as root.

Note that I've placed home.key in /root so that once the root partition is decrypted the home partition can be decrypted without requiring a second input of the password.

Now that the cryptab is setup, we need to rebuild the initramfs for boot. We do this by chroot-ing into the new system and running update-initramfs -u

# mount -o bind /proc root/proc
# mount -o bind /dev root/dev
# mount -o bind /dev/pts root/dev/pts
# mount -o bind /sys root/sys
# chroot root
# update-initramfs -u

And that's it. Reboot now and ubuntu will ask you for a password to expose the root partition before mounting it. You may notice a warning about /home not being mounted and given options to wait, skip, or do something else. Just give it a few more seconds. I guess the timeout built in for mounting /home isn't long enough to also deal with decrypting it.

USB Master Key

The process of entering the password every boot is troublesome and, to be honest, my ability to recall secure passwords is not to be trusted. In order to address this shortcoming, I decided to create a USB stick whose sole purpose was to act like a password. I do not consider this insecure because, should I lose the USB key, I can revoke the encryption passphrase from the LUKS header and make it no longer usable on my system.

I considered several options for how to go about creating the USB key. I considered a separate boot initramfs with grub installed on the key to actually boot from the key. I also considered storing the keyfile on a filesystem and searching removable filesystems at boot, or trying to identify the disk by partition label or UUID. Ultimately, I decided instead to use an unformatted partition of the USB key to store the keyfile.

The following can be performed on target system, however I suggest creating a backup of the boot/initramfs-…-generic file. If you mess anything up you can drop to grub (press shift 5 times at boot) and alter the grub boot command to use the backup initramfs to get back into a useable system.

First I created the keyfile as usual:

$ cat /dev/random | tee -a keymaster_root.key | key_progress.py
$ truncate -s 512 keymaster_root.key

Then I partitioned the USB key with an unformatted partition of 1MiB and the rest as LUKS/ext4. The second partition is somewhat irrelevant for this discussion except to show that the USB key is still useful for storage.

Now copy the keyfile to the unformatted space, and add the key to an unused slot of the LUKS header for the root partition

# dd if=keymaster_root.key of=/dev/sdb1 bs=1
# cryptsetup luksAddKey /dev/sdb1 keymaster_root.key

Now we find the unique name for the USB stick, particularly it's first partition so we can identify it when present:

$ ls -l /dev/disk/by-id | grep sdb1

In my case the USB key is given the path /dev/disk/by-id/usb- ADATA_USB_Flash_Drive_1411416161600078-0:0-part1. I then edited my crypttab as follows:

# /etc/crypttab
# root
root UUID=fdc69d8e-ede4-40f1-8a38-d69ba73c1c82 /dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1411416161600078-0:0-part1 luks,keyscript=/sbin/usb_keymaster

# home
home UUID=74c4c574-2396-434e-9f0f-ea9f8270c44e /root/nadie_chain_root_home.key luks

# swap
cryptswap /dev/sda6 /dev/urandom swap

The keyscript is a script which dm-crypt runs to get the passphrase. It expects the passphrase on stdout. I used the script from the stack overflow link above but took the initial parts of the script from the blog linked above (which makes the script work with plymouth).

The script attempts to read from the raw device provided in ${CRYPTTAB_KEY} (which is the third column in crypttab). If the device file does not exist after 5 seconds, it drops to asking for a password.

#!/bin/sh
#/sbin/usb_keymaster

# define counter-intuitive shell logic values (based on /bin/true & /bin/false)
# NB. use FALSE only to *set* something to false, but don't test for
# equality, because a program might return any non-zero on error
TRUE=0
FALSE=1

# set DEBUG=$TRUE to display debug messages, DEBUG=$FALSE to be quiet
DEBUG=$TRUE

# default path to key-file on the USB/MMC disk
KEYFILE=".keyfile"

# maximum time to sleep waiting for devices to become ready before
# asking for passphrase
MAX_SECONDS=2

# is plymouth available? default false
PLYMOUTH=$FALSE
if [ -x /bin/plymouth ] && plymouth -ping; then
    PLYMOUTH=$TRUE
fi

# is usplash available? default false
USPLASH=$FALSE
# test for outfifo from Ubuntu Hardy cryptroot script, the second test
# alone proves not completely reliable.
if [ -p /dev/.initramfs/usplash_outfifo -a -x /sbin/usplash_write ]; then
    # use innocuous command to determine if usplash is running
    # usplash_write will return exit-code 1 if usplash isn't running
    # need to set a flag to tell usplash_write to report no usplash
    FAIL_NO_USPLASH=1
    # enable verbose messages (required to display messages if kernel boot option "quiet" is enabled
    /sbin/usplash_write "VERBOSE on"
    if [ $? -eq $TRUE ]; then
        # usplash is running
        USPLASH=$TRUE
        /sbin/usplash_write "CLEAR"
    fi
fi

# is stty available? default false
STTY=$FALSE
STTYCMD=false
# check for stty executable
if [ -x /bin/stty ]; then
    STTY=$TRUE
    STTYCMD=/bin/stty
elif [ `(busybox stty >/dev/null 2>&1; echo $?)` -eq $TRUE ]; then
    STTY=$TRUE
    STTYCMD="busybox stty"
fi

# print message to usplash or stderr
# usage: msg  "message" [switch]
# command: TEXT | STATUS | SUCCESS | FAILURE | CLEAR (see 'man usplash_write' for all commands)
# switch : switch used for echo to stderr (ignored for usplash)
# when using usplash the command will cause "message" to be
# printed according to the usplash  definition.
# using the switch -n will allow echo to write multiple messages
# to the same line
msg ()
{
    if [ $# -gt 0 ]; then
        # handle multi-line messages
        echo $2 | while read LINE; do
            if [ $PLYMOUTH -eq $TRUE ]; then
                # use plymouth
                plymouth message -text="$LINE"
            elif [ $USPLASH -eq $TRUE ]; then
                # use usplash
                /sbin/usplash_write "$1 $LINE"
            else
                # use stderr for all messages
                echo $3 "$2″ >&2
            fi
        done
    fi
}

dbg ()
{
    if [ $DEBUG -eq $TRUE ]; then
        msg "$@"
    fi
}

# read password from console or with usplash
# usage: readpass "prompt"
readpass ()
{
    if [ $# -gt 0 ]; then
        if [ $PLYMOUTH -eq $TRUE ]; then
            PASS="$(plymouth ask-for-password -prompt "$1″)"
        elif [ $USPLASH -eq $TRUE ]; then
            usplash_write "INPUTQUIET $1
            PASS="$(cat /dev/.initramfs/usplash_outfifo)"
        else
            [ $STTY -ne $TRUE ] && msg TEXT "WARNING stty not found, password will be visible"
            /lib/cryptsetup/askpass "$1# echo -n "$1″ >&2
            # $STTYCMD -echo
            # read -r PASS /dev/null
            # [ $STTY -eq $TRUE ] && echo >&2
            # $STTYCMD echo
        fi
    fi
    echo -n "$PASS"
}

dbg STATUS "Executing usb_keymaster …"

# flag tracking key-file availability
OPENED=$FALSE

for TRY in 1 2 3 4 5
do
  if ! [ -e "${CRYPTTAB_KEY}" ]; then
    dbg TEXT "Waiting for USB stick to be recognized [${TRY}]"
    sleep 1
  fi
done # for TRY
if [ -e "${CRYPTTAB_KEY}" ]; then
  dd if="${CRYPTTAB_KEY}" bs=1 skip=0 count=512 2>/dev/nul
  OPENED=$TRUE
fi


# clear existing usplash text and status messages
[ $USPLASH -eq $TRUE ] && msg STATUS "                               " && msg CLEAR ""

if [ $OPENED -ne $TRUE ]; then
  # dbg TEXT "Failed to find suitable USB/MMC key-file …"
  readpass "$(printf "Unlocking the disk $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)\nEnter passphrase: ")"
else
  # dbg TEXT "Success loading key-file from $SFS ($LABEL)"
  # msg TEXT "Unlocking the disk $CRYPTTAB_SOURCE ($CRYPTTAB_NAME)"
  msg "Unlocking ${CRYPTTAB_SOURCE} (${CRYPTTAB_NAME}) from USB key"
fi

#
[ $USPLASH -eq $TRUE ] && /sbin/usplash_write "VERBOSE default"

Be sure to chmod +x /sbin/usb_keymaster to make the script executable.

In order to get this script into the initramfs we need to add the following hook file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/sh
#/etc/initramfs-tools/hooks/usb_keymaster

PREREQ=""

prereqs() {
  echo "$PREREQ"
}

case "$1″ in
  prereqs)
    prereqs
    exit 0
  ;;
esac

. "${CONFDIR}/initramfs.conf"
. /usr/share/initramfs-tools/hook-functions

copy_exec /sbin/usb_keymaster /sbin

Lastly, we need to ensure that the USB drivers are loaded at boot time (this may not be necessary with 14.04). Edit /etc/initramfs-tools/modules.

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
uhci_hcd
ehci_hcd
usb_storage

Lastly update the initramfs with

update-initramfs -u

Skype in 64bit Ubuntu 13.04


I've had some trouble getting skype to work in ubuntu 13.04. The first problem arose after switching to the nvidia binary graphics drivers. Skype kept trying to load the 64bit libraries instead of the 32bit compatability ones. This can be repaired by changing the linker's search path. The command to launch skype then is:

   ~$ LD_LIBRARY_PATH=/usr/lib32 skype

The second problem was that skype could not find pulse audio, so it was trying to look up hardware devices using ALSA (or so I've guessed by stdout). The solution is to install libpulse0:i386

   ~$ sudo apt-get install libpulse0:i386

After restarting skype my microphone and audio devices are all listed as "pulse audio" instead of having a dropdown for various options.

Also, as of skype 4.2.0.11 screen sharing appears to now be available in the linux version… finally.

Range Based for-loops in C++11


One of the new features of C++11 that is a complete win in my opinion is the support for range-based for-loop syntax. Judicious use allows for significantly more compact and readable code. However, one thing that is lacking from this feature is the ability to iterate over a range of integers. This isn't a problem, however, because it is very easy to implement.

// This file compiles as a single unit, you can test it by saving it 
// as range_for.cpp and compiling with 
// g++ -std=c++11 -o range_for range_for.cpp
// Adjust the interface as you see fit

// Implementation (put this in a header)
// --------------------------------------------


/// encapsulates a range of integral numbers for use in a c++11 range-based for 
/// loop
template< typename T>
struct Range
{
    /// the begin() and end() functions must return an iterator with a specific
    /// interface
    struct Iterator
    {
        T val;  ///< storage for the actual value

        /// implicit construction from the value type
        Iterator( T val ):val(val){}

        /// the range-based for loops attempt to dereference an iterator using
        /// this operator
        T operator*(){ return val; }

        /// the range-based for loops quit when the comparison of iter != end
        /// returns false
        bool operator !=( T other ){ return val != other; }

        /// iterators in a range-based for loop must have the prefix increment
        /// operator
        Iterator& operator++(){ ++val; return *this; }

        /// we likely want to implicitly convert to the value type
        operator T(){ return val; }
    };

    private:
        T m_begin;  ///< the first integral value
        T m_end;    ///< one past the last integral value

    public:
        /// construct a range [begin, end)
        Range( T begin, T end ):
            m_begin(begin),
            m_end(end)
        {}

        /// interface required by range-based for loop
        Iterator begin(){ return m_begin; }

        /// interface required by range-based for loop
        Iterator end()  { return m_end;   }
};


/// return an object which can be used in a range-based for loop
template 
Range range( T begin, T end )
{
    return Range(begin,end);
}




// Usage 
// ---------------------------------------------

#include 

int main(int argc, char** argv)
{
    std::cout << "Static ranges\n----------------\n";
    std::cout << "\nint:\n";
    for( int i : range(1,5) )
        std::cout << "   i: " << i << "\n";

    std::cout << "\nunsigned int:\n";
    for( unsigned int i : range(1u,5u) )
        std::cout << "   i: " << i << "\n";

    std::cout << "\nlong:\n";
    for( long i : range(1L,5L) )
        std::cout << "   i: " << i << "\n";

    std::cout << "\nDynamic range (program args)\n----------------\n";
    for( int i : range(0,argc) )
        std::cout << i << " : " << argv[i] << "\n";

    return 0;
}

The range<T>( T begin, T end ) function template returns a Range&ltT> object which supports the required interface for range-based loops. It has a begin and end method, both of which return an iterator. The iterator has the same storage says as the integral type used to instantiate the templates. It is implicitly convertable to and from the integral type. Futhermore, it has the prefix ++ operator, and can be "dereferenced" to get the stored integral value.

The implementation is likely to yield compiled code equivalent to a normal for loop.

The output of the demo program above is:

josh@Nadie:~/Desktop$ g++ -std=c++11 -o range_for range_for.cpp 
josh@Nadie:~/Desktop$ ./range_for arg1 arg2 arg3 arg4 arg5
Static ranges
-----------

int:
   i: 1
   i: 2
   i: 3
   i: 4

unsigned int:
   i: 1
   i: 2
   i: 3
   i: 4

long:
   i: 1
   i: 2
   i: 3
   i: 4

Dynamic range (program args)
-----------
0 : ./range_for
1 : arg1
2 : arg2
3 : arg3
4 : arg4
5 : arg5
josh@Nadie:~/Desktop$

Making a list of everything you've installed with apt-get


As a clean install of ubuntu is often the only reliable way to upgrade, it's useful to know everything that you've installed manually. You could get a list of everything that's marked in for installation with something like:

dpkg -get-selections | grep -v deinstall > my-selections

but that list tends to be very long and it includes all the dependencies. So if you install a meta package which depends on a specific version of some other package you'll be trying to install that specific version after upgrade, which is not what we want. In that case all we want is the meta package.

Luckly, apt-get keeps a history of every time it is called. These history files are in /var/log/apt/history.log.(\d).gz. Each entry in that list contains a "Commandline: …" line which lists the command line that was executed. Thus we can get a list of everything we've installed by grepping through these files.

The first script is a perl script which parses "Commandline: …" entries:

#!/usr/bin/perl
# parse_apt_log.pl
while(<>)
{
    if(/^Commandline: apt-get install(.+)/)
    {
        unless(/-reinstall/)
        {
            print join("\n", split(/\s+/, $1) );
        }
    }
}

and the second is a bash script which pipes the contents of those history files to the perl script

#!/bin/bash

outfile="$(mktemp)"

gunzip -c /var/log/apt/history.log.*.gz | ./parse_apt_log.pl > $outfile

sort "$outfile"
rm "$outfile"

Which gives an output like:

josh@Nadie:~/Desktop$ ./make_pkg_list.sh
astyle
bitcoin-qt
debathena-pharos-support
dhex
digikam
docbook2x
docbook5-xml
docbook-xsl-ns
firefox
flip
fontforge
fop
gdm
git-svn
gnucash
gyp
icedtea6-plugin
icedtea-7-plugin
jhead
latexml
libapache2-mod-fastcgi
libavcodec-extra-53
libav-dbg
libavdevice-extra-53
libavfilter-extra-2
libavformat-extra-53
libavutil-extra-51
libcairo2-dbg
libcrypto++-dev
libcurl4-gnutls-dev
libfcgi-dev
libgdal1-dev
libglade2.0-cil
libgnome-desktop-dev
libgtkglext1-dev
libjpeg-turbo-progs
libjson0-dev
libmosquitto0
libpcre3-dev
libqtsysteminfo1
libtelepathy-qt4-2
libtidy-dev
libusb-0.1-4
libusb-1.0-0-dev
libusb-dev
libxml2-dev
libxml2-utils
linux-generic
markdown
meld
mtpfs
mtp-tools
network-manager-openvpn
okular
openjdk-7-jre
owncloud-client
pcsxr
pcsxr
pdfedit
python-rosinstall
qt-sdk
qtsixa
ragel
ros-fuerte-desktop-full
ros-fuerte-pr2-*
ros-fuerte-pr2-simulator
ros-fuerte-rqt
rtmpdump
sixad
smplayer
sound-juicer
spawn-fcgi
sqlitebrowser
synapse
ufraw
ufraw-batch
uncrustify
universalindentgui
untex
vlc-dbg
xdotool
xmlstarlet
xmlto
xsltproc

You can reinstall this list on the new system with sudo apt-get installtr '\n' ' ' < package-list.txt``, where tr will remove the newlines.

Other install notes for upgrading to ubuntu 13.04:

To install deb packages from the command line along with their dependencies install gdebi-core from the repositories. Then run, for instance sudo gdebi google-chrome-stable-current_amd64.deb.

To remove overlay scrollbars: gsettings set com.canonical.desktop.interface scrollbar-mode normal

To allow rawdisk vmdk's in Virtualbox the user must be a member of the disk group: sudo usermod -a -G disk user

C++ FreeFontStack


I've been doing a lot of research into free software font technologies and as a consequence I've been playing around a lot with Freetype2 and Fontconfig (actually the library interface libfontconfig). These are "straight up" c-libraries and so I (personally) find the interface a bit "clunky" in C++ code. Don't get me wrong, I wouldn't presume to criticize their design as it is clear the authors/maintainers are far more knowledgeable about this stuff than I am… but from a users perspective I desire a bit more.

So… I've written some C++ wrappers for freetype2 and fontconfig. The main goals in these projects are:

  • Don't change the usage patterns of the library
  • Provide a pointer-like type which automatically performs reference counting for reference counted objects
  • Provide a pointer-like type which exposes member-functions where relevant
  • Provide a pointer-like type which still allows access to the public data members of the underlying structures
  • Allow access to the native c-pointer for use in parts of the library that I haven't finished wrapping

So far… cppfontconfig wraps more or less all the functionality of libfontconfig. I'd still like to add some interface for assignment by touples in order to grab error codes when desired without the error being part of the function signature. Perhaps in the future.

cppfreetype isn't nearly as complete (freetype2 is a much larger library). I still haven't decided the appropriate way to deal with structures which are normally stack allocated. I do however have wrappers for FT_Face, FT_GlyphSlot, and an iterface for iterating over contours and points in an FT_Outline.

These libraries are released as GPLv3. I have public git repositories and publicly accessible doxygen documentation available in the links below. No bug tracker though since I'm giving up on redmine.

Tweaking JabRef in Ubuntu


A couple of months ago I switched from using Mendeley to JabRef. JabRef is a lot less pretty and lacks a lot of the features of Mendeley, but it a much more solid piece of software. I particularly love that it works with a native bibtex file as a backend, so I can keep my bib file under version control, check it out as a git-remote/svn:external in my latex projects, and have a use the same cite keys in all my documents.

There were a couple of annoyances that I had with JabRef that I managed to work around so I figured I would document them here.

Look and Feel

The "native" java look and feel has all the aesthetic bells and whistles of a 90s era unix desktop, and drives me nuts. Luckily, jabref allows you to specify the look and feel class in it's advanced preferences. Got to options->preferences->advanced and check "Use other look and feel". I use the Gnome desktop so I set this to com.sun.java.swing.plaf.gtk.GTKLookAndFeel and now Jabref looks like it belongs.

JabRef with GTK look and feel

Tiny Treeview for Groups

I like to organize my references (rather than simply search for them) to help me find multiple references for a similar topic. JabRef, however, limits you to only 10 or so rows in the treeview for groups. I found this to be rather frustrating. Unfortunately, you can't change the number of rows dynamically (by clicking and dragging) but there is an undocumented preference entry which sets the number of visible rows. Start by going to options->preferences and the clicking "export preferences" to export a preference file for JabRef with all your current preferences. Save it as a .xml file. Now open that with your favorite text editor and add the following entry within <node name="jabref"><map></map></node>

<entry key="groupsVisibleRows" value="30″/>

Feel free to change "30″ to whatever you like. Now the groups treeview will be a bit larger and more useful. Also, take a look at all the other preferences which you didn't know jabref had and feel free to play around with them to tune your jabref experience.